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ABSTRACT 



ANALYZING THE IMPACT OF A DATA ANALYSIS PROCESS 
TO IMPROVE INSTRUCTION USING A 
COLLABORATIVE MODEL 

Rebecca B. Good, EdD 
Texas A&M University-Commerce, 2006 

Adviser: Sherion Jackson, EdD 

The Data Collaborative Model (DCM) assembles assessment literacy, reflective 
practices, and professional development into a four-component process. The sub- 
components include assessing students, reflecting over data, professional dialogue, 
professional development for the teachers, interventions for students based on data 
results, and re-assessing to measure the impact of the changes made in both teacher 
practice and student interventions. The need to study the impact of the DCM initiative 
necessitated this study. 

The purpose of this quantitative, causal-comparative study was to examine the 
relationship between using the Data Collaborative Model (DCM) and student 
achievement through TAKS passing rates in math and reading. The rationale was 
campuses that have attempted to create and implement a culture of data-driven decision 
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making in a collaborative, reflective setting over a 3-year period of time would see an 
increase in teacher effectiveness (DuFour, 2004b). This process should have a natural 
output of increased student performance (Stiggins, 2004). Particularly, this study 
examined whether or not this was true in the six high implementation schools selected 
after comparing their TAKS reading and math passing rate means against six campuses 
that had low implementation of the DCM process and tools based on survey results. 

The study revealed that there was not a statistical significance in TAKS passing 
rates in reading between high and low implementation campuses in the within and 
between tests. There was a statistical difference, however, in math, both in the within and 
the between tests in Usage Type. The study also showed that high implementation 
campuses had higher TAKS passing rates and mean gains than low implementation 
campuses. Finally, it revealed that teachers are more likely than administrators not only 
to understand the DCM, but to use the process and tools more often. 

In conclusion, this study found that further research would be required to analyze 
why there was a statistical difference in math but not reading. It also recommended that a 
similar study should follow in 2 years to re-measure the effectiveness of the Data 



Collaborative Model. 
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Chapter 1 



INTRODUCTION 

Children are the living messages we send to a time we will not see. 

— N. Postman (1982, p. xi) 

All around the nation, school reform is in the air (Fullan, 2001). Whether it is 
through accountability issues or high school redesign, reform is now a constant inside and 
outside educational venues. Educational issues frequently make the front pages of daily 
newspapers and are discussed by those who understand school finance issues, property 
taxes, and resulting connections to the state legislature, especially in Texas. 

In the “old days” teachers were lone rangers. They wrote their lesson plans alone 
and closed their door when they were teaching (Combs, Miser, & Whitaker, 1999; 
Martin-Kniep, 2004). Although they might share ideas when a new teacher asked for 
discipline help, usually that was the extent of teacher collaboration. 

In the present educational system, the No Child Left Behind Act (NCLB) of 2001 
(National Conference of State Legislatures, 2006), has come to dominate the direction of 
education. NCLB legislation called for stronger instructional practices aligned to 
assessment systems that could be counted on to close the very noticeable achievement 
gap between children of poverty and certain ethnic groups and Anglo students. The 
purpose of NCLB clearly stated that all children were to have an equal opportunity to 
acquire a high quality education that would be measured by academically challenging 
state assessments. 

Educational journals such as Phi Delta Kappan, Educational Leadership, Journal 
of Staff Development, School Administrator, Journal of Curriculum Studies, and others 
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have brought their readers promising practices that have worked in schools around the 
country. Practices such as constructivist versus direct instruction, collaborative lesson 
studies, professional learning communities, job-embedded professional development, 
team meetings, smaller learning communities, data-driven decision making, and 
distributed leadership have all been studied, debated, written about, and initiated on 
campuses all over the country (Council of the Great City Schools, 2004; DuFour & 

Baker, 1998; Hunt, 2005; Klonsky, 1998). Some of the better known researchers 
espousing school reform practices include Larry Lezotte (1991) with the effective school 
correlates, Michael Fullan (1998) who talked about cultures of change, Rick Stiggins 
(2004) who wrote extensively on assessment literacy, Rick DuFour (2004b) with his 
professional learning communities, Robert Marzano (2003) with classrooms that work, 
and Mike Schmoker (2004b) and Peter Senge (1990) who have written at length on 
learning organizations and what it takes to change them. Schmoker (2004b) has begun to 
write forcefully about what is working in successful schools: 

Thousands of schools and even entire districts can attest to the power of these 
structures for promoting first incremental and then cumulatively dramatic and 
enduring improvements in teaching and learning. (...) schools and districts have 
made substantive, enduring gains in achievement, largely on the strength of well- 
structured, goal-oriented learning teams and communities, (p. 431) 

The necessity to increase the quality of American public education has been well 
documented since the 1980s when A Nation at Risk was published (National Commission 
on Bxcellence in Education, 1983). Achievement scores for high school students were the 
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lowest in two decades, and SATs were down as well (National Education Goals Panel, 
2000). The National Commission on Excellence in Education recommended looking at: 

. . .the quality of teaching and learning in all educational areas, which include 
private and public school and institutions of higher learning; comparing American 
education against other advanced nations; looking at the relationship between 
high school graduation and college entrance along with which educational 
programs lead to success in college; looking at causal factors that have impacted 
education in the last twenty-five years; [and] defining the problems in education 
which must be overcome if the nation is to have excellence in education, ('ll 2) 

The only surprise 23 years later is how little has been achieved in the area of 
educational reform. This statement is supported by the fact that the NCEB goals seem to 
be a repeat of the goals set by the National Goals Panel in the 1980s (Center for 
Educational Decision Support Systems, 2004). The National Goals Panel goals, which 
were to address areas such as the quality of teaching and learning in all educational areas, 
high school graduation rates, and college entrance rates, remain current issues. 

Statement of Problem 

The age of accountability (Guskey, 1998) has focused attention on not only what 
is taught but also how it is taught (Marzano, 2003). Not only is there a need for reliable 
assessments aligned to rigorous curriculum but also how to use the information given by 
the assessments (Stiggins, 2004). A process must be in place to help campuses 
understand how to read data and to turn the information into an action plan: 

When this process is absent, confusion reigns. Staff from (...) schools indicated 
that they did not see the connection among teacher-administered in-class 
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assessments, their norm-referenced district test, and the large-scale state 
assessment. Nor did they know what to do with this information. (Learning Point 
Associates, 2000, p. 1) 

Although some states are expanding their databases and using more reliable 
accountability systems, most have not begun to promote the need for local capacity in 
understanding the data received (Center for Educational Decision Support Systems, 

2004). Administrators and teachers have been handed pages of data reports for several 
years without much, if any, instruction in what to do with the information. As the reports 
get more sophisticated, the data can show in an even deeper manner what skills students 
are not performing well. Information on utilizing data to impact classroom instruction in 
order to increase student achievement has been lacking. 

Purpose of the Study 

The purpose of this study was to examine the relationship between using the Data 
Collaborative Model (DCM), a process of instructional improvement using data analysis, 
and student achievement through TAKS passing rates in math and reading. The focus was 
on whether the DCM process and tools changed teacher practice enough to impact 
student achievement. The rationale was that campuses which have attempted to create 
and implement a culture of data-driven decision making in a collaborative, reflective 
setting over a 3-year period of time, would see an increase in teacher effectiveness. This 
process should have a natural output of increased student performance. The study 
examined whether or not this was true in the six high implementation schools selected. 



based on survey results. 
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Research Questions and Hypothesis 

The research questions were written in collaboration with district officials who 
had an interest in the results of the study. Of particular interest to this study were the 
questions regarding whether the data analysis process in place for 3 years actually had an 
impact on student achievement on those campuses having a high level of implementation 
of the Data Collaborative Model system. The following research hypothesis and 
questions were used to guide this study: 

Hypothesis: There is a statistically significant difference in TAKS passing rates 
between high implementation and low implementation campuses after using the Data 
Collaborative Model for a 3-year period of time. 

1 . What was the relationship for teachers between the self-reported (based on the 
DCM survey results) understanding versus actual use of common and 
continuous assessment? 

2. What impact does regularly scheduled teacher reflection on data have on 
student achievement? 

3. What is the relationship between job-embedded professional development and 
student achievement? 

4. What impact does the use of the DCM have on student achievement over a 
period of 3 years? 

5. Does exposure to a model that uses data analysis to impact instruction result 
in a change in the usage of the district’s online data web? 

6. Is there a difference in DCM usage perception between campus administrators 



and teachers? 
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7. What impact does the principal have on the implementation of the DCM 
model? 

Description of the Study 

The Data Collaborative Model (DCM) assembled assessment literacy, reflective 
practices, and professional development into a four-component process. The sub- 
components include assessing students, reflecting over data, professional dialogue, 
professional development for the teachers, interventions for students based on data 
results, and re-assessing to measure the impact of the changes made in both teacher 
practice and student interventions. The need to study the impact of the DCM initiative 
necessitated this study. The study helped the district with the funding decision related to 
implement the DCM initiative. 

This research study was a causal-comparative study that looked at student 
achievement over a 3-year period of time between two groups of campuses. The first 
group, called the high implementation group, stated through their survey responses that 
their campus understood and used the DCM process and tools at a high level based on 
means. The second group, called the low implementation group, consisted of campuses 
that had a lower level of understanding and usage of the DCM process and tools. 

A field test was conducted using the DCM survey created to measure the impact 
of this initiative. The pilot was sent out to a sample of 20 people purposely selected to 
mirror the categories listed on the survey. The categories were principals, deans, content 
teachers, instructional specialists, special education and ESL teachers, fine arts teachers, 
and Technical Assistance Providers (TAP). When the pilot surveys were returned, the 
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feedback and data were analyzed to see if changes to the instrument were warranted. A 
more in-depth discussion of the field test can be found in Chapter 3. 

The participants in this study were sent a revised survey (see Appendix A) based 
on the pilot study analysis. The survey requested that administrators and teachers answer 
questions about the degree of implementation of the process and the various DCM tools 
available. Returned surveys identified the campus by numbers and campus administrators 
and teachers by position, not by name. The survey questions helped identify six campuses 
that had a high degree of DCM implementation over the last 3 years and six campuses 
that had a low level of implementation. Once those 12 campuses were selected, a 
database was constructed. The database included reading and math passing rates from the 
Texas Essential Knowledge and Skills (TAKS) test over the 4-year period that the TAKS 
had been given. Also added to the database were additional DCM trainings that each 
campus had received. The control treatment was that all campus principals had received a 
standard training on the DCM process and tools over the last 3 years. Additional training 
opportunities were available to those sub-areas and/or campuses that requested them. 

The district studied was the second largest in the state of Texas with an urban 
population of 160,000 students. The ethnic breakdown was 61% Hispanic, 31% African 
American, 6% White, and 1% Other (Texas Education Agency, 2004). The district 
encompasses an area of 312.6 square miles and includes all or portions of 1 1 
municipalities (Wikipedia, 2006, 'll 2). The district’s students came from homes where 
almost 70 different languages were spoken and over one-third of the student population is 
classified as second language learners (Texas Education Agency, 2004). More than 79% 
of the district’s students qualify for free/reduced lunch assistance. 
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Urban issues including poor student achievement, inexperienced teaching staff, 
political strife, high student mobility, a lack of instructional rigor and coherence, and 
bureaucratic business practices can be seen in this district just like in other urban districts 
(Council of the Great City Schools, 2002). Over the past 3 years, many initiatives were 
implemented to help create a stronger change climate. Among those initiatives were 
process management training for central office and other interested campus personnel; 
cross-functional training; Balanced Scorecard (Kaydos, 2003); Lean Six Sigma (Six 
Sigma, 2006); realignment of the written, taught, and tested curriculum (English, 2000); 
and finally, instructional improvement by means of data analysis through the Data 
Collaborative Model (Garcia, 2005). 

Assumptions of the Study 

To further explain this study, several assumptions were made. The first 
assumption was that the information reported by the participants on the survey was 
honest and accurate. Second, the TAKS data information received from the Texas 
Education Agency’s Academic Excellence Indicator System (AEIS) was accurate. The 
third assumption was that all the data turned into the Texas Education Agency was 
accurate. In addition to this was the assumption that other data on the Texas Education 
Agency website was also accurate. The final assumption is that most administrators and 
teachers want positive educational change, even if they are unsure how to go about 
implementing the change (Eullan, 2001). 

Limitations 

This study was limited in several ways. The first limitation was that principals 
who received the e-mailed request to pass the survey link on to their teachers did not 
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always do so. The campus staffs that received the survey did not always take the time to 
answer it, which was the second limitation. A lack of serious approach by the campuses 
in the use of the three early release days as professional learning communities 
opportunities was a third limitation. The fourth limitation was based on teacher turnover, 
principal changes, and sub-area feeder pattern changes that occurred during the 3-year 
study. A final limitation was that the researcher was held to the timelines of the 
leadership of the district as to when approval was granted to send out the survey to the 
principals and that the survey would be classified as optional by the district leadership. 
Significance of the Study 

The Data Collaborative Model has been in place for 3 years. This study was 
designed to examine the effectiveness of the model that used the teaching practices of 
meeting regularly with student data and sharing successful strategies in order to change 
instruction. As campuses put time and energy into implementing best practices, it was 
important to measure the effectiveness of those practices (Killion, 2002). 

Campuses must keep a careful and timely eye on what is an effective instructional 
practice and what is not. Accountability measures such as Adequate Yearly Progress 
(AYP) and the Academic Excellence Indicator System (AEIS) publish the grades of each 
campus and district yearly using the results of state assessments as one component of the 
school evaluation system. Campus and district leaders must choose the most effective 
practices based on research to help impact student results and close achievement gaps 



(Elmore, 2005; Eullan, 1998). 
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Definitions of Terms 

The following is a list of supported definitions. The terms should help clarify 
meaning throughout the remainder of the study. 

Academic Excellence Indicator System (AEIS)-“pulls together a wide range of 
information on the performance of students in each school and district in Texas” every 
year based on indicators such as the Texas Assessment of Knowledge and Skills (TAKS) 
test, drop out and attendance rates, and more (Texas Education Agency, 2002, 'll 1). 

Adequate Yearly Progress (AYP)-the process NCEB uses to represent “the 
annual academic performance targets in reading and math that the State, school districts, 
and schools must reach to be considered on track for 100% proficiency by school year 
2013-14” (Colorado Department of Education, 2004, The ABCs of AYP page, 'll 5). 

Assessment literacy-“the collective capacity of teachers and leaders in schools to 
examine data, make critical sense of it, develop action plans based on the data, take 
action and monitor progress along the way” (Eullan, 2001, p. 127). 

Benchmark assessment-designed to assess student mastery as a result of Texas 
Assessment of Knowledge and Skills (TAKS) instruction and serve as an indicator for 
student success on TAKS (Garcia, 2005). 

Collaborative/reflective-an environment where teachers meet together in a shared 
setting to reflect over data such as assessments, student work, and lesson plans (Cowan, 
2006; Eerraro, 2000). 

Distributed leadership-leadership as spread across multiple people and situations 



(Timperley, 2005). 
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Job-embedded professional development-instructional learning that teachers 
acquire as they go through their daily work activities. It can include discussion with 
others, peer coaching, mentoring, study groups and action research in both a both formal 
and informal manner (U.S. Department of Education Professional Development Team, 
1994). 

Professional Learning Communities (PLC)-a meeting where teachers “focus on 
learning rather than teaching, work collaboratively, and hold [each other] accountable for 
results” (DuFour, 2004b, p. 6). 

Researcher and Researcher Bias 

The researcher has been the Executive Director of the Data Collaborative Model 
for the past 2 years. The researcher created many of the tools and processes used in the 
trainings. The researcher understands that she may have a bias toward wanting the Data 
Collaborative Model to be a successful change process for campuses. Although 
principals’ and teachers’ perceptions will form the basis of all conclusions, the researcher 
also has assumptions that may influence the interpretation of results. 

Summary and Chapter Overview 

The urban student tends to be a minority child living in poverty (Council of the 
Great City Schools, 2002). Achievement gaps are the norm for this student. He or she has 
a higher chance of being impacted by second language, special education, and/or single 
parent issues. Too many students are being left behind while ways to teach them are 
sought (English, 2006; Marzano, Pickering, & Pollock, 2001). As Fenwick English said 
in 2006 at a conference for Texas administrators: “without alignment [between 
assessments and curricula], student and community demographics will continue to be the 
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largest predictors of student test results— noted since 1951 by Edward L. Thorndike in the 
research literature” (English, 2006). The urban child needs teachers and administrators 
who understand how to balance demographics with a high level, standards-based 
curriculum that is relevant and rigorous. As mentioned by Eenwick English, this is the 
only way campuses will be able to avoid having demographics predict student 
achievement. 

Over the last two decades, great strides have been made to add more rigor to the 
national and state standards (National Governors Association, 2006). Alignment between 
what is taught and what is tested has been strengthened, but there is a long way to go 
before there is a closing of the achievement gap (National Governors Association, 2006). 
Research has identified best practices that have proven effective with urban students 
(Marzano, 2003; Resnick & Glennan, 2002; Schlechty, 2002; Schmoker, 2004a, 2004b). 
The research says a campus must have consistent implementation of these best practices 
by teachers of urban students using a collaborative, reflective model such as the DCM 
(DuEour & Eaker, 1998; Eerraro, 2000; Martin-Kniep, 2004). 

Chapter 2 contains a review of the literature related to the historical background 
of educational reform and those policies that led to educational change in the last 30 
years. The components of the Data Collaborative Model, which include assessment 
literacy, professional learning communities, job-embedded professional development, 
distributed leadership, and organizational change are summarized with a focus on the 
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Chapter 2 



REVIEW OE THE REEATED EITERATURE 



The best practices that made up the Data Collaborative Model included 
assessment literacy, professional learning communities, and job-embedded professional 
development. Two necessary practices that were also included in the model were 
distributed leadership and organizational change. The historical perspective will set the 
tone for the need for an initiative with practices that were in the DCM. 

Historical Perspective 

School reform has been a practice in the United States since the mid- 17th century. 
The first educational movement in the New World began with the Puritans in 1630, who 
came not to bring religious freedom for all, but to enforce the religion in which they 
believed (Henson, 2003). According to Henson, “The main motivation for the Puritans’ 
long, expensive, and dangerous journey to the New World was their desire to build a new 
community to worship God” (p. 85). Because the Puritans thought the Devil would “take 
advantage of their children’s ignorance and mislead them” (p. 85), they quickly moved to 
establish schools. The first were the Dame Schools, where only reading was taught at 
home, usually in the kitchen. Eater the Old Deluder Satan Act, passed in 1647, required 
communities with more than 50 families to teach reading and writing, which continued to 
take place in the setting of the home. Once the community reached 100 or more families, 
a more formal Eatin school was added to the community. In a Eatin school, the religious 
and language -based curriculum had to be taught by a hired teacher, not the families 
themselves (Henson, 2003). 



13 




14 



By the mid- 18th century, the Franklin Academy, with its more practical 
curriculum, which included the teaching of mathematics, bookkeeping, and astronomy, 
replaced the Latin school curriculum (Henson, 2003). A century later, during the 
“common school movement” (Webb, Metha, & Jordan, 2000, p. 175), half of the nation’s 
children were in school (Henson, 2003). According to Webb et al. (2000), “State control 
as well as direct taxation for the support of the common schools-publicly supported 
schools attended in common by all children-became accepted practices” (p. 176). 
Educational change had already begun and would continue until present time: 

The history of American education is replete with examples of supposed 
panaceas. Taking the shape of “reforms,” these well-meaning efforts-often 
worthy in themselves-have dotted the educational landscape since the time of 
Horace Mann in the 1830s and 1840s. For example, the common school itself 
would remove all crime and poverty from American society. Mann described it as 
the “greatest invention ever made by man.” (Hunt, 2005, pp. 84-85) 

In the late quarter of the 19th century, secondary public school became more 
firmly established (Webb et ah, 2000). The secondary public school movement had 
grown slowly before then, with most of the secondary schools being located in the North. 
It was not until industrial growth “intensified the demand for skilled workers” (p. 183) 
that secondary high school growth increased. High schools became necessary not only to 
meet the skill needs of the industrial age, but were also important to help the children of 
the large number of immigrants learn about “American ideals and values” (p. 183). 

In 1892, the National Education Association (NBA) saw a need to standardize the 
curriculum (Webb et ah, 2000). It established a Committee of Ten, which made 
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recommendations to schools that they include vocational training and practical courses in 
their curricula. They asked for, “an early introduction to the basic subjects and uniform 
instruction and instruction for both college -bound and terminal students, with few 
electives” (p. 184). It was this committee that also created the concept that courses should 
meet four to five times a week for a year. A unit of credit would then be earned after 
completion of the year. 

The American comprehensive high school, as we know it today, was essentially in 
place by the 1920s. It had been extended to 4 years and typically had students aged 14-18 
who attended (Resnick & Hall, 1998; Webb et ah, 2000). Junior high schools began 
appearing in urban areas, “as an outgrowth of the recommendation by the Committee of 
Ten that academic work begin earlier and that elementary schooling be reduced from 
eight to six years” (Webb et ah, 2000, p. 185; see also Resnick & Hall, 1998). 

Many educational changes occurred in the 20th century due to social, economic, 
and political factors. Population growth was up to 106 million (up from 50 million in 
1880) (Population Reference Bureau, 2006), due in part to upwards of half a million 
immigrants arriving each year from 1905-1914 (Webb et ah, 2000). The country was also 
becoming increasingly more urban. The economic growth spurred by the industrialization 
of the country created rapid growth in railroad and other transportation industries as well 
as communication industries. With this came an increase in the number of students 
attending school which in turn led to a need for more schools, more teachers, and 
certainly more resources. By the end of World War II, returning servicemen filled the 
colleges, and within a few years the postwar baby boom hit the public schools (Webb et 



ah, 2000). 
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By 1957, the Sputnik scare motivated the federal government to pass the National 
Defense Act (NDEA). The NDEA “was instituted primarily to stimulate the advancement 
of education in science, mathematics, and modern foreign languages” {Columbia 
Encyclopedia, 2001, National Defense Education Act, 'll 1). More curriculum reforms 
followed, each recommending that rigor be increased in the academic core areas (Henson, 
2003). 

It was not until the third quarter of the 20th century that there would be any 
substantial improvement in the education of African Americans, Native Americans, 
Mexican Americans, and Asian Americans (Webb et ah, 2000). Separate schools and 
poor resources were the norm for these students until the civil unrest of the 1950s and 
1960s (Resnick & Hall, 1998). 

In 1983, the National Commission on Excellence in Education published its 
report, A Nation at Risk: The Imperative for Educational Reform (National Commission 
on Excellence in Education, 1983). It was one of many such reports that year, but its title 
and provoking language set it apart almost immediately. The book’s message warned that 
the country was sliding into educational mediocrity that endangered the nation's future 
(Gordon, 2003). Several risk indicators delineated the upcoming threat in that it 
compared American students with other industrialized nations and did not fare well. It 
also predicted that the College Board's Scholastic Aptitude Tests (SAT) and other 
College Board achievement tests would reveal consistent declines in recent years in such 
subjects as physics and English. (National Commission on Excellence in Education, 

1983) A Nation at Risk became the symbolic opening barrage in a 20 year struggle to 
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improve schools, as the report helped put education reform at the top of the national 
agenda. 

In 1989, the National Education Goals Panel (NEGP) was created during the 
National Education Summit in Charlottesville, North Carolina. This was an 
unprecedented meeting between President George H. W. Bush and the nation’s 
governors, including Bill Clinton, governor of Arkansas. It was the first time people in 
these positions had met together to focus exclusively on how to improve the educational 
performance of the nation (The National Education Goals Panel, 2000). The Summit led 
to the adoption of six (later eight) National Education Goals that were to be met by 2000. 
The goals stated that 

All children will start school ready to learn, high school graduation will increase 
to 90%, students will become competent in challenging subject matter, teachers 
will have the knowledge and skills they need, and U.S. children will be first in the 
world in mathematics and science. (National Education Goals Panel, 2000, Goals 
Work! f 5) 

Whereas the result of previous educational initiatives looked for results in 
monitoring compliance with rules and regulations, the National Education Goals Panel 
focused on results (National Education Goals Panel, 2000). This changed the way 
educational systems were judged. By 2001, 41 states began issuing annual report cards on 
campus performance. Almost half the states began identifying high and low performing 
schools by this time as well. Before the National Education Goals Panel was established, 
educational reform movements did not last through changes in leadership either at the 
national, state, or local levels (The National Education Goals Panel, 2000). In 1991, at the 
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recommendation of The National Goals Panel, Congress established the National Council 
on Educational Standards and Testing. This bipartisan council looked at establishing 
national standards to describe what students needed to know and be able to do. The 
council recommended that national standards be established that had high expectations, 
were dynamic and not static, and not be federally mandated. Grants were then provided 
through federal agencies and private sources to encourage the development of national 
standards. By 2000, most states had established statewide standards that were more 
rigorous in the four core areas and had begun to align assessment systems to them 
(National Education Goals Panel, 2000). 

In 1989, the National Education Goals Panel had concluded that there were not 
data available to compare states easily, there were no commonly acceptable performance 
levels, no comparable high school completion information, and no comparable state data 
on school safety issues such as violence or drugs. Once these gaps were identified, 
collection efforts increased at all levels in all areas (National Education Goals Panel, 
2000 ). 

When President Clinton came to office in 1991, he adopted most of the 
recommendations that came out of the first Education Summit. He then created his own 
proposal, called Goals 2000, which produced the National Education Standards and 
Improvement Council (National Conference of State Eegislatures, 2006). This Council 
could approve or reject states' standards. During the midterm elections of 1994, many 
people in Congress voiced strong resistance to the increased federal role in education and 
the council consequently died. But the increased accountability and federal focus on 
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standards continued despite the opposition (National Conference of State Legislatures, 
2006). 

The government continued its engagement in accountability issues when 
President Clinton signed the Improving America's Schools Act (IAS A) of 1994 (National 
Conference of State Legislatures, 2006). IAS A was a reauthorization and revision of the 
original Elementary and Secondary Education Act (ESEA). This legislation asked for 
continued development of state standards, development of categorized performance 
standards (partially proficient, proficient, and advanced for each content area), 
implementation of assessments aligned with instruction in grades 3-12, measurement of 
Title 1 students, and 

. . .the use of performance standards to establish performance standards to 
establish a benchmark for improvement referred to as “adequate yearly progress.” 
All Eocal Education Authorities (EEA) and schools must show continuous 
progress toward meeting the AYP standard or face possible consequences, such as 
having to offer supplemental services and school choice or replacing the existing 
staff. (National Conference of State Eegislatures, 2006, History page, 'll 15) 
Although states were allowed to create their own educational system, the states had to 
follow the requirements of lASA to be able to continue receiving federal funds (National 
Conference of State Eegislatures, 2006). 

Enter into this climate of educational focus on standards and assessment the 
reauthorization of Elementary and Secondary Education Act (ESEA), now called the No 
Child Eeft Behind Act (NCEB of 2001). This Act, signed by President Bush in January of 
2002, increased federal funding to states by more than 24%. Attached to the increase in 
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federal funding came a rise in federal mandates and sanctions. The act required those 
states that accepted the federal funds to increase student testing and to improve the 
quality of the assessments as well. It also mandated that schools have highly qualified 
teachers in each classroom as defined by NCLB. This act also held schools accountable 
for the performance of students with strong penalties for not meeting Adequate Yearly 
Performance (AYP) requirements (National Conference of State Legislatures, 2006). The 
age of accountability, as Thomas Guskey (1998) and others (Goldhaber & Hannaway, 
2004) have called it, was and continues to be, upon the nation. 

This historical review sets the tone for the current educational environment of 
reform. Education related books and journals are filled with studies of instructional and 
other school-related practices that increase teacher practice and student effectiveness 
(DuFour, 2004b; Marzano, 2003; Marzano, Pickering, & Pollock, 2001; Schmoker, 
2004a). The Data Collaborative Model took several of the best practices that researchers 
had studied and written about and folded them into an initiative to help campuses impact 
instruction. 

Accountability Practices 

When talking about impacting instruction through a data analysis process, one 
first must break it up into the practices or components that make up the process. The 
components of the Data Collaborative Model are a combination of (a) assessment 
literacy; (b) professional learning communities, which includes collaborative and 
reflective practices; (c) job-embedded professional development; (d) distributed 
leadership; and (e) organizational change. All of these components, when implemented 
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on a campus, pave the way for a collaborative culture that leads to assessment literacy 
(Fullan, 2001; Timperley, 2005). 

Rick Stiggins (2004), along with other researchers (Bernhardt, 1999; DuFour & 
Faker, 1998; Holcomb, 2004; Schmoker, 2004b; Stiggins, 2004), all suggested 
integrating the curriculum with assessment and creating an assessment culture. In 
addition to learning how to align assessment to their instruction, campus teachers have 
also been asked to examine how instruction is provided (DuFour, 2004b; Henson, 2003; 
Jacob, 2004; Marzano, 2003; Marzano, Pickering, & Pollock, 2001; Schlechty, 2002; 
Schmoker, 2004a). Educational researchers have studied methods that increase student 
performance and teacher effectiveness (Allington & Cunningham, 1996; DuFour 2004a, 
2004b; Marzano et ah, 2001; Schlechty). Among the most written about “best practices” 
are professional learning communities (DuFour & Baker; Hord, 1997; Schmoker, 2004a, 
2004b), collaboration over student work and teacher- made tests (DuFour & Baker), 
checking for rigor and relevancy in the every day curriculum (Erikson, 2002; Jacob, 
1997, 2004; Klonsky, 1998), distributed leadership (Timperley, 2005), job-embedded 
professional development (DuFour, 2004a; U.S. Department of Education Professional 
Development Team, 1994), and learning organizations (Schmoker, 2004a, 2004b; Senge, 
1990; Senge, Kleiner, Roberts, Ross, & Smith, 1994). 

Assessment literacy. In this age of accountability, it has been found through 
research studies that not all administrators and teachers understand the relationship 
between what is written, taught, and tested (Earl & Fullan, 2003; English, 2000). As 



Eorna Earl and Michael Fullan wrote in 2003, 
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There was a time in education when decisions were based on the best judgment of 
the people in authority. It was assumed that school and district leaders, as 
professionals in the field, had both the responsibility and the right to make 
decisions about students, schools and even about education more broadly. They 
did so using a combination of intimate and privileged knowledge of the context, 
political savvy, experience and logical analysis. Data played almost no part in 
decisions, (p. 383) 

The need to understand the role of assessment is becoming more important as 
school personnel come to understand that it is a vital piece of an instructional process that 
includes data-driven decision making (Earl & Fullan, 2003). Assessment literacy for 
administrators and teachers is still a relatively new field. In 1999, Rick Stiggins 
mentioned during an interview with Dennis Sparks from the Journal of Staff 
Development, that only three states required competence in assessment for principal 
certification (Sparks, 1999). Few teachers and administrators had an opportunity to 
develop assessment literacy skills. A teacher spends one third to one half of his or her 
professional time in assessment-related activities, so it is necessary to acquire skills to 
know how to use assessments as instructional tools (Sparks, 1999). 

Schmoker (2004b) wrote that only in learning teams can assessment literacy, 
which is so necessary to the ongoing improvement of instruction, be obtained. 

Workshops will not allow “the application of and experimentation with new assessment 
ideas in real classrooms, and sharing that experience with other colleagues in a team 



effort” (Schmoker, 2004b, p. 430). 
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Professional learning communities. Much is being written on the need for district 
professionals to understand better the relationship between assessment and curriculum 
(Learning Point Associates, 2000; National Education Association (NEA), 2003; 
Schmoker, 2004a; Stiggins, 2004). Testing what was just taught does not always happen 
in a clearly defined way. Much has to do with the teachers’ lack of training in how to 
create reliable assessments (Sparks, 1999; Stiggins, 2004). 

The literature has begun to connect the data with action plans that lead to a 
change in instruction and professional development (Center for Educational Decision 
Support Systems, 2004; Eearning Point Associates, 2000; NEA 2003; Stiggins, 2004). 
Professional learning communities have been one of the mechanisms used to achieve this 
through encouraging teachers to discuss data and to share strategies in an effective 
manner (DuEour, 2004b). Others besides DuEour, the originator of the practice, have 
begun to favor the use of professional learning communities (PEC): 

Eearning communities are the means by which we can break [teacher] (...) 
isolation and foster a collaborative environment and reflective culture. In fact, 
creating a collaborative environment has been described as “the single most 
important factor” for successful school improvement initiatives and the “first 
order of business” for those seeking to enhance the effectiveness of their school. 
(Eastwood & Eewis, 1992, p. 215) 

Shirley Hord (1997), a long time advocate for educational change and 
professional learning communities, wrote that both teachers and students benefit from 
teachers meeting as learning communities. The benefits included reduction in the 
isolation of teachers, increased commitment to the mission, shared responsibility for the 
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development of student and their success, increased meaning and understanding of the 
content that teachers teach, and a higher likelihood that teachers were well informed, 
professionally renewed, and inspired to inspire students. Teachers also felt more job 
satisfaction, had higher morale, and less absenteeism. The benefits for students were as 
necessary as the ones for teachers: 

The benefits for students included decreased dropout rate and fewer classes 
"skipped," lower rates of absenteeism, increased learning that is distributed more 
equitably in the smaller high schools, greater academic gains in math, science, 
history, and reading than in traditional schools and smaller achievement gaps 
between students from different backgrounds. (Hord, 1997, p. 28) 

In his more recent articles, Schmoker (2004a, 2004b) reiterated the need for and 
lack of use of best practices such as PLCs. Schmoker was clear in what the research is 
saying about teacher practice: 

Mere collegiality won't cut it. Even discussions about curricular issues or popular 
strategies can feel good but go nowhere. The right image to embrace is of a group 
of teachers who meet regularly to share, refine and assess the impact of lessons 
and strategies continuously to help increasing numbers of students learn at higher 
levels. (Schmoker, 2004a, 'll 12) 

Schmoker also listed researcher after researcher who agreed that the collaborative 
practice emphasized by PLCs must happen to have an increase in student achievement: 
The concurrence on [continuous, structured teacher collaboration] is both 
stunning and under-appreciated. Advocates for focused, structured teacher 
collaboration include Roland Barth, Emily Calhoun, Linda Darling-Hammond, 
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Richard Elmore, Michael Fullan, Bruce Joyce, Judith Warren Little, Dan Lortie, 
Milbrey McLaughlin, Fred Newmann, Susan Rosenholtz, Rick Stiggins, James 
Stigler, Joan Talbert, Gary Wehlage, Grant Wiggins, Ronald Wolk and numerous 
others. (Schmoker, 2004a, p. 1) 

The research was clear, the studies on successful practices more numerous, and 
yet the results were the same. Elmore found in 2005 that campuses were reluctant to 
embrace practices that work due to internal and external factors. An internal factor such 
as a lack of accountability would make schools less successful for the following reasons: 
They are less likely to exercise control over student curriculum and performance. 
They also are more likely to grasp for superficial solutions to external pressures- 
for example, teaching test items rather than developing and teaching higher-level 
content. And, they are more likely to set expectations in the prevailing culture of 
atomization and the existing abilities of individuals-for example, pushing harder 
on individual teachers instead of designing collective responses that make the 
work of the organization more powerful. (Elmore, 2005, p. 2) 

Fullan (2005) insisted that PECs, as done in the present educational system, 
would not gain beyond 20% in popularity in a district and would have only pockets of 
strong usage. The practice would only last as long as those that have sustained its use 
were on that campus unless a greater perspective was understood and added. Fullan wrote 
of a need for tri-level implementation, where the PECs’ concept was brought to the 
attention of not only the school community but also the district and state level as well. 
Without all those entities supporting the use of PECs, a systemic change would not occur. 
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Job-embedded professional development. A veteran teacher’s definition of 
professional development would more than likely be, a set number of hours in staff 
development done in order to fulfill a contractual obligation for employment, usually 
done through district or local service center offerings. The literature now challenged that 
thinking (DuFour, 2004a; Resnick & Glennan, 2002; U.S. Department of Education 
Professional Development Team, 1994). As early as 1994, the U.S. Department of 
Education promoted job-embedded professional development as “continuous inquiry and 
improvement embedded in the daily life of schools” (U.S. Department of Education 
Professional Development Team, 1994, The Mission & Principles of Professional 
Development page, 'll 5) and “planned collaboratively by those who will participate in and 
facilitate that development” ('][ 5). As Rick DuEour wrote, “The traditional notion that 
regarded staff development as an occasional event that occurred off the school site has 
gradually given way to the idea that the best staff development happens in the workplace 
rather than in a workshop” (2004a, p. 63). 

One of the key strengths of the professional teaching and learning cycle was its 
design as a job-embedded professional learning process that was ongoing and results 
driven. According to multiple correlation studies on teacher quality (Darling-Hammond, 
2000; Darling-Hammond et ah, 2003), higher levels of student achievement were 
associated with educators who participated in sustained professional development 
grounded in content- specific pedagogy. Continuous professional learning that increased 
teacher outcomes, in turn, impacted student outcomes. Reviews of studies indicated that 
when teachers improved their instructional practices, student achievement also improved 
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(Darling-Hammond, 2000; Marzano et al., 2001; Southwest Educational Development 
Laboratory, 1997). 

Organizational change. The literature focused on the need to create a school 
climate that was ready to accept change. Michael Fullan (1998, 2000, 2001, 2005) has 
written much in this area as well as Shirley Hord (1997). Peter Senge (1990), in his The 
Fifth Discipline, wrote that one must master the three components of organizational 
change: design, structure, and implementation. The desired change must get the whole 
organization “engaged and committed, both in favor of a shared vision and in a rigorous 
search for the truth. (...) Any coercive process, no matter how well intended, simply 
cannot result in commitment” (Senge, 1990, p. 438). 

Distributed leadership. The literature on collaboration, professional learning 
communities, and reflective practices all mention that a natural outcome of these actions 
is teacher empowerment. As mentioned by Garcia in 2005, “Distributed leadership has its 
basis in cognitive development theory. Russian scholar Lev Vygotsky’s theory of 
cognitive development pointed out the link between social interaction and the 
development of individual intelligence” (pp. 26-27). In essence, individual intelligence 
grows due to biological factors when people engage in a physical and social (cultural) 
environment (Timperley, 2005). 

Today’s principals are being asked to share their leadership with teachers for a 
more effective collaborative school culture (DuFour & Baker, 1998; Hord, 1997). 
Sergiovanni (1996) explained that the foundation of authority for leadership was rooted 
in shared ideas, not in the power of position. 




28 



Summary and Chapter Overview 

Chapter 2 contains a review of the literature related to the historical background 
of educational reform and those policies that led to educational change in the last 30 
years. It also includes a description of the components of the Data Collaborative Model 
(DCM). The components of the DCM, which includes assessment literacy, professional 
learning communities, job-embedded professional development, distributed leadership, 
and organizational change are summarized with a focus on the leading researchers. 

Chapter 3 contains a description of the methodology used to study the effects of 
the DCM components on 12 campuses according to their level of usage over a 3-year 
period of time. Sample selection procedures, instrument development, data gathering, and 
data analysis procedures are discussed. The objective of this quantitative study was to 
study the impact of the DCM, an instructional improvement process through data analysis 



model, on student achievement. 




Chapter 3 



METHOD 



This chapter provides a description of the quantitative design used in the current 
study. The description of the subjects and the participating campuses appear first. The 
next two sections explain the data collection and procedures used. Then the construction 
and use of the databases and how data were analyzed follow. 

The aim of educational research is to develop new learning about educational 
phenomena (Borg & Gall, 1989). Even further, the purpose is to build confidence that 
certain claims about the educational phenomena being studied were true or false. In the 
case of quantitative research, the goal is to gather data that prove or disprove the 
knowledge claim (Borg & Gall, 1989). 

Quantitative research in education can be categorized as one of two types: 
descriptive studies and studies intended to discover causal relationships (Borg & Gall, 
1989). Descriptive studies deal with finding out “what is,” and the causal-comparative 
method “is aimed at the discovery of possible causes for the phenomena being studied by 
comparing subjects in whom a characteristic is present with similar subjects in whom it is 
absent or present to a lesser degree” (p. 331). 

This research study is a causal-comparative study, looking at student achievement 
over a 3-year period of time between two groups of campuses. The first group of 
campuses, called the high implementation group, had stated through their survey 
responses that their campus understood and used the DCM process and tools at a high 
level based on mean survey responses. The second group of campuses, called the low 
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implementation group, was made up of campuses that had a lower level of understanding 
and usage of the DCM process and tools, also based on survey results. 

The DCM has been a district initiative for 3 years. The district itself was 
interested in measuring the impact of the DCM on student achievement. The researcher 
and district officials from the Evaluation and Accountability department collaborated on 
survey questions (see Appendix A) that could meet the needs of both the district and the 
researcher. Although most of the questions were of vital interest to the researcher, there 
were a few questions whose answers had little or no impact on this study. For example, 
survey item 22 asked if the participant knows whom to contact with a question about the 
DCM. The question was necessary for the district to know, but of minimal impact on this 
study. 

Subjects 

Over 1,000 campus administrators and teachers from a large southern urban 
public school participated in the survey. There were 217 campuses in the district, 
including 37 high schools, 24 middle schools, and 156 elementary schools. The schools 
were divided into six sub-areas, with a sub-area superintendent responsible for each sub- 
area. An e-mail from the deputy superintendent went out to the six sub-area 
superintendents asking if they would volunteer their campuses to participate in a survey 
of the Data Collaborative Model process and tools (see Appendix B). Sub-area 
superintendents e-mailed back their response to the deputy superintendent who then 
forwarded the e-mails to the researcher. Out of six sub-areas, five agreed to participate. 
That limited the campuses who could take the survey from 217 to 162 campuses. 
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Participating Campuses 

Of the 162 schools that had the option to take part in the survey, 121 campuses 
returned surveys, resulting in a participation rate of approximately 75%. There were 12 
elementary campuses that were selected for inclusion in the study. The 12 were selected 
first because there were more elementary schools that responded to the survey and also 
their staff participation rates were higher as well. The second manner of narrowing the 
field to 12 campuses was by high and low DCM process and tools usage rate based on 
mean results for survey items 3, 6, and 7. This allowed the researcher to use a sample of 
campuses (i.e., 10% of campuses that participated in the survey) to represent the entire 
district. Through analysis of DCM survey results, six campuses were determined to have 
high levels of understanding and usage of the DCM process and tools and were compared 
to six campuses that were judged to have a low level of DCM understanding and usage. 

A more thorough explanation is provided in the Procedures section of this chapter. 

Data Collection 

The survey instrument consisted of 22 questions with an estimated 7 to 10 
minutes completion time (see Appendix A). The survey instrument was created in 
collaboration with the district’s Evaluation and Accountability researchers. Permission 
was granted by district officials for use of their materials. The collaboration between the 
district and the researcher ensued because both sought information from this study. The 
survey was made available through an e-mailed invitation containing a web address to the 
survey, rather than a hard copy sent via the U.S. mail, since web surveys have been found 
to have as successful a response rate as ones sent through the mail (Kieman, Kieman, 



Oyler, & Gilles, 2005). 
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Also, the survey was made available through a link on the district data website 
(see Appendix C). Notice of the survey web address was sent to campus administrators 
through district e-mail three times to encourage participation (see Appendix D). Each 
time, a request was made of the principals to forward the DCM link to campus teachers. 

The survey was written to identify 12 campuses that had high and low levels of 
usage of the Data Collaborative Model (DCM). Specifically, this survey examined how 
well teachers and campus administrators understood and used the DCM process and 
tools. The survey also was used to determine the difference between administrator and 
teacher usage perception. 

Prior to administering the survey, a panel of professional colleagues examined the 
survey instrument for content validity. Minor changes in wording were incorporated 
when necessary. A copy of the informed consent letter and the survey can be found in 
Appendix E and A respectively. 

TAKS passing rates for 2003-2006 were collected from a district database for the 
12 campuses chosen to be studied. The TAKS reading and math passing campus rates 
represent a combination of all grades and students tested in each subject area. 

Data on web usage by the campuses for 2004 through 2006 were also collected 
from the Office of Institutional Research (OIR). Those data were used to measure 
growth, if any, by campuses of data web usage. Web usage is measured by the number of 
pages visited. 

Enrollment data and other district information were gathered through various 
outside sources so that district confidentiality could be maintained. Additional DCM 
training given to the 12 campuses was added to the data gathered. This was formulated 
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into an intensity level and was used to help analyze whether additional DCM trainings 
impacted a campus’ identification as high or low implementer. 

Procedures 

A field test was conducted using the initial survey instrument. In a field test, “all 
or some of the survey procedures are tested on a small scale that mirrors the planned full- 
scale implementation” (National Center for Education Statistics, 2006, Glossary page, 

'll 8). It was sent to a purposefully selected sample of 20 people. Included within that 
sample were principals, deans, content teachers, special education and ESL teachers, fine 
arts teachers, and Technical Assistance Providers (TAP). When the pilot surveys were 
returned, the data were analyzed to see if changes to the instrument were warranted. 
Minor revisions were made and the final survey was sent to the deputy superintendent of 
Instructional Services for approval. 

After receiving permission from the deputy superintendent to proceed with the 
DCM survey (see Appendix E), a request was sent from that same office to sub-area 
superintendents to ask if they would be interested in taking part in the survey (see 
Appendix B). Out of the six sub-areas, five responded to the request. The DCM survey 
online link was e-mailed to the principals and deans of instruction of the schools in the 
five areas that responded positively to the request (see Appendix C). The five areas 
comprised of 162 schools that could choose to take part in the survey. The e-mail 
requested that principals and deans make the survey link available to their campus 
teachers. The survey link was available both through e-mail and on the DCM data web 



page. 
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A 14-day time window was allocated to complete the survey. A second e-mail 
was sent to sub-area superintendents to remind them of the survey and to ask for their 
support by encouraging their campuses to complete it. The informed consent letter was 
also attached to the e-mail (see Appendix E). The letter stated that, by returning the 
completed survey, consent was assumed. Confidentiality was maintained and protected 
by the researcher. The records were kept by the researcher in a locked file and in a non- 
networked computer. 

After the window of time had closed, campus response was checked. An e-mail 
reminder with a deadline extension of two more days was sent to campuses that had not 
responded. A third e-mail was sent out to remind principals of the survey (see Appendix 
D). At the end of the survey period, 121 campuses out of a possible 162 (75%) had 
responded with at least one participant. Of those 121 campuses, 92 (76%) were 
elementary, and 29 (24%) were secondary schools. 

When the survey data on campus participation were assessed, it was decided that 
campuses would be selected from the elementary level because of their high response rate 
of 92 schools (76%). Purposeful selection techniques were used to identify the 12 
elementary campuses. Campuses were then identified as high and low implementers (six 
each) based on the following criteria: (a) percentage of staff numbers participating in the 
survey and (b) mean number of choices for survey items 3, 6, and 7. 

The selection began by sorting the database of respondents by the percent of staff 
who took the survey and then by the number of staff who took the survey. A campus with 
35% participation and 31 staff members taking the survey was determined to be more 
suitable for inclusion in the study than a campus with 43% participation and 17 staff who 
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participated, because the study was interested in having the participation of as many 
campus professionals as possible. 

In addition to the percent of campus staff who participated, levels of 
implementation of DCM were assessed. Six campuses with high levels of DCM 
implementation and six with low levels of implementation were selected. Levels of 
implementation were determined by looking at survey items 3, 6, and 7 by campus. The 
three items measured (a) the participants’ understanding of the DCM tools and process, 
(b) the use of the DCM tools and process, and (c) whether they felt the DCM process had 
an impact on their student achievement. Survey items 3, 6, and 7 listed data analysis tools 
and processes that were recommended and reinforced through DCM training. The 
number of choices selected for each question were aggregated by school and a mean 
number of choices was computed. 

Demographic data were reported for the 12 schools (see Table 1). This included 
total enrollment and percentages of (a) students receiving free- or reduced-priced lunch, 
(b) limited English proficiency (LEP), (c) Talented and Gifted (TAG), and (d) special 
education students. Ethnic composition was also reported. 

The student demographic data on each of the 12 campuses selected for this study 
are shown in Table 5. Out of the 12 campuses, 10 were 58% or more Hispanic, and the 
other two had a more congruent composition of White, African American, and Hispanic 
students. Of the Hispanic campuses, 6 of the 10 were 47% or more limited English 
proficient (EEP). All 12 campuses had a special education population between 1% and 



9%, all within the expected range. 




Table 1 



Demographic Data for the 12 Selected Campuses 





Total 

Enrollment 


Free 

Lunch 

% 


White 
# % 


Af. Am. 
# % 


Hispanic 
# % 


LEP 

# 


% 


TAG 

# 


% 


Special ED 
# % 


High 1 


993 


90 


2 


1 


36 


4 


954 


96 


630 


64 


54 


6 


38 


4 


High 2 


623 


95 


13 


2 


43 


7 


540 


87 


381 


62 


31 


5 


51 


9 


High 3 


471 


63 


175 


38 


125 


27 


162 


35 


56 


12 


37 


8 


5 


1 


High 4 


487 


83 


9 


2 


191 


40 


279 


58 


139 


29 


78 


16 


33 


7 


High 5 


203 


97 


0 


0 


42 


21 


161 


80 


58 


29 


0 


0 


13 


8 


High 6 


690 


97 


9 


2 


33 


5 


633 


92 


376 


55 


74 


11 


52 


8 


Low 1 


767 


92 


15 


2 


92 


12 


660 


86 


332 


44 


29 


4 


46 


6 


Low 2 


705 


85 


11 


2 


25 


4 


659 


94 


345 


49 


35 


5 


39 


6 


Low 3 


831 


89 


3 


1 


5 


1 


821 


99 


388 


47 


64 


8 


50 


6 


Low 4 


552 


86 


68 


13 


77 


14 


404 


74 


194 


36 


57 


11 


44 


8 


Low 5 


702 


76 


320 


46 


79 


12 


300 


43 


127 


18 


40 


6 


43 


7 


Low 6 


814 


97 


9 


3 


24 


13 


781 


96 


401 


50 


60 


8 


29 


4 



o^ 
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When analyzing the data, survey questions were grouped in three ways. The first 
way was the selection of high and low implementation campuses. The survey questions 
were used in the selection of campuses by using the number of participants and the 
percentage of participating staff members and by the mean number of total options 
chosen on survey items 3, 6, and 7. Once the high participation campus data were sorted, 
the six high and six low implementation campuses were selected by looking at the means 
of the total options chosen by participants. 

The second manner was by subsets of the survey questions that emerged based on 
the themes (see Table 2). The factor themes were (a) understanding the DCM process and 
tools, (b) implementation of the process and tools, (c) collaborative/reflective practice, 
and (d) professional development. In addition to the original seven research questions 
established at the onset of this project, three research sub-questions were created to 
investigate the factor themes generated from the survey responses. The three factor 
research sub-questions also asked the following: 

1. What was the mean score for each factor by school and type of respondent? 

2. Can using mean scores on factors predict TAKS passing rates or mean scale 
score by factors grouping all 12 campuses together? 

3. Did low and high implementation schools score differently on the factors? 
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Table 2 

Survey Questions that Aligned with Each Factor Before Cronbach’s Coefficient Alpha 
was Computed 





Understanding of 
the DCM Process 
& Tools 


Implementation of 
the Process & Tools 


Collaborative/ 

Reflective 

Practice 


Professional 

Development 


Survey 

Questions 


4, 6, 10, 11, 12 


3, 5,7,9, 13, 19,21 


14, 15, 16 


8, 17, 18,22 



A Cronbach alpha was run to assess internal consistency reliability on the factors. 
Cronbach's alpha measures how well a set of items (or variables) measures a single 
unidimensional latent construct. Technically speaking, Cronbach's alpha is not a 
statistical test, it is a coefficient of reliability (or consistency). Cronbach’s alpha can be 
written as a function of the number of test items and the average inter-correlation among 
the items (UCLA Academic Technology Services, 2006). 

When internal consistency reliability was computed on the four factors, the data 
showed that some items were not appropriately placed. Alphas were examined, and 
several factors were altered based on reliability statistics and the corrected item-total 
correlation. Factors were revised until Cronbach’s alpha was adequate at 0.7 or above. 
Table 3 shows the revised configuration of items to factors. 
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Table 3 

Survey Questions that Aligned with Each Factor After Cronbach’s Coefficient Alpha was 
Computed 





Understanding of 
the DCM Process 
and Tools 


Implementation of 
the Process & 
Tools 


Collaborative/ 

Reflective 

Practice 


Professional 

Development 


Survey 

Questions 


10, 11, 12 


5, 9, 13, 19, 20 


4, 14, 15, 16 


8, 17, 18,21 



The third way to analyze the data was by the seven original research questions, 
which were answered by specific survey questions. They were answered after the 
selection had been made of the six low implementation campuses and the six high 
implementation campuses. 

A hypothesis was also stated. The researcher hypothesized that there was a 
statistically significant difference in TAKS passing rates between high and low 
implementation schools after using the Data Collaborative Model for a 3-year period of 
time. 



Research Question 1 : What is the relationship for teachers between the self- 
reported (based on the survey results) understanding versus actual use of common and 
continuous assessment? High and low implementation schools were compared using 
Cross Tabulations to assess differences in survey responses 4, 6, and 7, which asked 
whether teachers understood and actually used the DCM tools. The rationale was that if 
teachers understood common and continuous assessments, then they would have a high 



level of usage of the DCM tools. 
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Research Question 2: What impact does regularly scheduled teacher reflection on 
data have on student achievement? This was answered by regressing the responses from 
survey item 20, the Collaborative and Reflective factor scores, and Usage Type on high 
and low TAKS reading and math percent passing rate changes from 2003 to 2006. The 
Collaborative and Reflective factor and Usage Type were the independent variables and 
TAKS growth was the dependent variable. 

Research Question 3: What is the relationship between job-embedded 
professional development and student achievement? The Professional Development 
factor score was used along with Usage Type to predict TAKS passing rates using 
regression analysis to answer this question. The Professional Development factor and 
Usage Type were the independent variables and TAKS growth was the dependent 
variable. 

Research Question 4: What impact does the use of the DCM have on student 
achievement over a period of 3 years? Survey items 14, 16, 17, and 20 were used to 
answer this question. Responses to these items were used in a repeated measures 
ANOVA to observe potential relationships with TAKS passing rates. The survey items 
were the independent variable and the dependent variables were TAKS gains in passing 
from 2002 to 2003, 2003 to 2004, and 2004 to 2005. 

Research Question 5: Does exposure to a model that uses data analysis to impact 
instruction result in a change in the usage of the district’s online data web? This question 
was answered by analyzing the 2 years (2004-2006) of available web usage data by 
campus. Frequency of usage for high and low implementation schools were contrasted 



using percentages. 
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Research Question 6: Is there a difference in DCM usage perception between 
campus administrators and teachers? Survey items 3, 6, 7, 8, 9, 13, 15, 16, 19, and 20, 
which compared administrator to teacher answers on the same campus, were used to 
answer this question. Differences in mean principal ratings and teacher ratings were 
compared using a multiple regression with the ratings as the dependent variable and rater 
type (principal or teachers) and implementation level as the independent variables. Effect 
sizes were determined using Cohen’s effect size statistics:/ for regression analysis and 
q for analysis of variance. These values reflect the proportion of variance in academic 
indicators explained by the independent variable(s) divided by the proportion of variance 
attributed to error (R2/1-R2), or how much of the variance in DCM usage perception 
between teachers and administrators can be explained by DCM tools and processes 
(Newton & Rudestam, 1999). There are only “rules of thumb” as to what values make an 
effect size of practical significance. Cohen (1988) suggested values of/^ = 0.02 as small, 
0.15 as medium, and 0.35 as large effect sizes, or = 0.10 as small, 0.25 as medium, and 
0.40 as large effect sizes. 

Research Question 7 : What impact does the principal have on the implementation 
of the DCM model? This question was answered by comparing survey items 6 and 7 
selections between the principal and his or her campus teachers. Those items measure the 
amount of understanding and usage of the data analysis tools on their campus. If teachers 
and principals each checked five of the eight choices (63%), although perhaps not the 
same five choices, schools were judged as having both understanding and usage. 
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DCM Database 

Once the 12 campuses that were to be studied were identified from the survey 
responses, a database was created. The database contained the following information: 

1. Campus demographics, including total campus enrollment, socio-economic 
level, ethnicity, limited English proficiency (LEP), gifted (TAG), and special 
education status were identified in the database. 

2. The sub-area to which the campus was assigned (including if the campus had 
changed areas within the last 2 years) was labeled. This was important 
because each sub-area made individual decisions regarding what DCM 
training they would allow. Some sub-areas did not allow any, while some 
asked for repeated sessions. Eeeder pattern changes occurred three times in the 
3 years of this study. 

3. Principal turnover rate was included because DCM implementation varied by 
principal. 

4. Additional DCM trainings given to campuses over the last 2 years were 
recorded. Although the DCM process began in the 2003-2004 school year 
with the introduction of the benchmark template, the DCM process was not 
fully developed with continued professional development. In the fall of 2004, 
a position was created to help implement DCM at a district level. 

5. DCM Intensity level was calculated and recorded. Eirst, the number of extra 
DCM training hours a campus had received over and above the standard 
training given to all district principals was calculated. This number was 
multiplied by the intensity factor. An intensity factor was assigned to each 




43 



type of training, taking into account who on the campus received the training. 
If a whole campus received training, it was rated higher than if just the teacher 
leaders received training. The rationale was that the more campus staff who 
received the training, the stronger the possibility that it would be 
implemented. All day sessions with follow up, such as INOVA training, were 
given the highest intensity factor rating. 

6. TAKS reading and math passing rates-Spring 2003 to 2006 reading and math 
were gathered in order to compare the first year without DCM to the other 
three with DCM available to the campuses. 

7. Data web usage by campus was added to see if there would be a correlation 
between campuses that use the DCM and their usage of the data web. 

Data Analysis 

Data were analyzed to provide a general picture of campus understanding and 
usage of the district’s data analysis process and tools. The following explains the 
different types of questions found on the survey. 

Survey item 2 asked each participant to identify himself or herself as a principal, 
dean, assistant principal, instructional specialist. Technical Assistant Provider (TAP), or 
teacher. Question 2 was used to create respondent comparison groups to be the 
independent variable in a one-way analysis of variance. Anonymity was kept by having 
the teachers fill in only their campus number on their survey and not their names. 
Teachers were asked to identify more specifically whether they were general education, 
ESL, special education, or fine arts teachers. The district used this to measure the 
involvement of different teacher groups in the data analysis process. 
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For multiple response questions, the following coding system was used: 1 = first 
answer, 2 = second answer and 3 = third answer, and so forth. Only survey items 3, 6, 
and 7 required this coding system used specifically to measure the frequency and percent 
of each response. The remaining questions asked for a single response. 

Survey items 4 and 8 through 24 used a six-point Likert scale format. These were 
coded with 1 = 1, 2 = 2, 3 = 3, 4 = 4, 5 = 5, and 6 = 6. The frequency and percent of each 
response, as well as the mean, was computed. 

Summary and Chapter Overview 

Chapter 3 explained the design and methods of this quantitative study. The focus 
of the study was also stated. Data gathering and data analysis techniques were discussed. 
The survey instrument was described along with how it was sent out to the participants 
and how it was analyzed when the responses were returned. The database containing 
different types of data was also described along with how it was used. 

When analyzing the data, survey questions were grouped for analysis in three 
ways. The first way was the selection of high and low implementation campuses. The 
second way was by factors, created from subsets of survey questions. The third way was 
by research questions which were answered by which were answered by specific survey 
questions and factors. 

Chapter 4 contains the results of the data analyses answering the questions posed 
by the way the survey questions were grouped. Various tables and figures display the 
statistical results of how the schools were selected, statistical results from the four 
factors, and the answers for the seven research questions. 




Chapter 4 



RESULTS 



The purpose of this study was to examine the relationship between using the Data 
Collaborative Model (DCM), a process of instructional improvement using data analysis, 
and student achievement at 12 campuses in a large urban district. The rationale was that 
campuses, which have attempted to create and implement a culture of data-driven 
decision making in a collaborative, reflective setting over a 3-year period of time, would 
see an increase in teacher effectiveness and student achievement. There were six high 
implementation campuses and six low implementation campuses selected to be studied in 
this causal-comparative study based on a survey sent to 162 campuses spread over five 
district sub-areas. 

The research questions were written in collaboration with school district officials 
who had an interest in the results of the study. Of particular interest to this study were the 
questions regarding whether the data analysis process that had been in place for 3 years 
had an impact on student achievement on those campuses that have a high level of 
implementation of the system. 

When analyzing the data, survey questions were grouped for analysis in three 
ways. The first way was the selection of high and low implementation campuses. The 
second way was by factors, created from subsets of survey questions. The factors chosen 
were (a) understanding the DCM process and tools, (b) implementation of the process 
and tools, (c) collaborative/reflexive practice, and (d) professional development. There 
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were three factor research sub-questions included as well. The third grouping was by the 
seven research questions. 

High and low implementation campuses were selected based on two criteria: (a) 
the number and the percentage of staff members participating in the survey and (b) the 
mean number of total options chosen for survey items 3, 6, and 7. More consideration 
was given to those campuses with a higher number of participating staff versus a high 
percentage of participating staff. Percentages of participating staff participation rates can 
be found in Table 4. The high implementation campuses were labeled as High 1 to High 
6. The low implementation campuses were labeled Low 1 to Low 6. 

Survey items 3, 6, and 7 measured whether participants thought the DCM had an 
impact on student achievement, whether they understood the components, and finally 
which tools they actually used during the DCM process, respectively. The means for 
campus were computed for each question (see Table 5). The six high implementation and 
six low implementation campuses were selected by first finding the largest participant 
campus groups that answered the survey, then looking at the means for each of the three 
questions. 

The high implementation campuses had means between 4.8 and 6.5 for survey 
item 3, between 2.5 and 4.4 for survey item 6, and between 2.4 and 3.7 for survey item 7. 
The low implementation campuses had means between 2.3 and 5.6 for survey item 3, 
between 0.9 and 2.8 for survey item 6, and between 0.9 and 2.4 for survey item 7. 
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Table 4 



Staff Participation Rates 





Teachers 




Participating Staff 


by Campus 


# 


% 


High 1 


78 


22 


28.2 


High 2 


48 


19 


39.6 


High 3 


33 


19 


57.6 


High 4 


43 


16 


37.2 


High 5 


21 


21 


100.0 


High 6 


62 


21 


33.9 


Low 1 


52 


15 


28.8 


Low 2 


41 


13 


31.7 


Low 3 


50 


17 


34.0 


Low 4 


33 


17 


51.5 


Low 5 


38 


37 


97.4 


Low 6 


58 


16 


27.6 
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Table 5 

Total Options Chosen on Survey Items 3, 6, and 7 by Campus 



Survey Item 3: 
DCM has Impact 



Survey Item 6: 
Knowledge 
of DCM Tools 



Survey Item 7 : 
Use the DCM Tools 





# 


Mean 


# 


Mean 


# 


Mean 


High 1 


144 


6.5 


63 


2.9 


58 


2.6 


High 2 


91 


4.8 


66 


3.5 


54 


2.8 


High 3 


98 


5.2 


57 


3.0 


45 


2.4 


High 4 


92 


5.8 


62 


3.9 


59 


3.7 


High 5 


100 


4.8 


52 


2.5 


42 


2.0 


High 6 


107 


5.1 


92 


4.4 


62 


3.0 


Average-High 




5.3 




3.4 




2.8 


Low 1 


79 


5.3 


36 


2.4 


31 


2.1 


Low 2 


44 


3.4 


31 


2.4 


24 


1.8 


Low 3 


96 


5.6 


47 


2.8 


40 


2.4 


Low 4 


49 


2.9 


35 


2.1 


24 


1.4 


Low 5 


164 


4.4 


83 


2.2 


58 


1.6 


Low 6 


36 


2.3 


14 


0.9 


14 


0.9 


Average-Low 




3.9 




2.1 




1.7 



Note. N = total options chosen. 
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Factors 

The survey items were grouped into four factors, and Cronbach alpha was 
calculated for each. When the alphas were run, not all the items within the factors proved 
reliable. The factors were then revised, and new alphas were computed. The revised 
factors are found in Table 3 in Chapter 3. 

The Understanding factor assessed how well persons surveyed understood the 
DCM process. The Understanding factor had a Cronbach alpha of 0.759, near the desired 
range of 0.80 (Leech, Barrett, & Morgan, 2005), and all three Corrected Item-Total 
Correlation items measured between 0.388 and 0.737, indicating that all the items were 
good components. 

The Implementation factor measured whether campuses implemented the DCM 
process and tools. The alpha coefficient measured 0.739, which indicates reasonable 
reliability, with all the Corrected Item-Total Correlation items at acceptable levels of 
between 0.370 and 0.588. 

The Collaborative/Reflective factor measured whether practitioners felt like they 
were taking part in collaborative and reflective practice on their campus. The Cronbach 
alpha measured 0.736, indicating adequate reliability, and all the Corrected Item-Total 
Correlations items were between 0.328 and 0.671. 

The Professional Development factor asked educators whether they received 
professional development by discussing strategies at the collaborative table and by 
looking at data. The alpha was 0.479 on this factor, which showed inadequate reliability 
when survey Item 21 (I feel I need more training on the OIR Data Web) was included in 
the factor. The Cronbach alpha increased to 0.726 when survey item 21 was deleted. 
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After the data were grouped in this manner, the three research factor sub- 
questions were answered. Their results follow. 

Factor Research Sub-Question 1: What was the mean score for each factor by 
school and type of respondent? The campus means by factors demonstrate that the high 
implementation campuses were higher in each factor than the low implementation 
schools. The mean averages of each implementation type reflect that as well. The higher 
implementation campuses had a stronger understanding of the Data Collaborative Model 
and were more likely to use DCM (see Table 6). The campus mean for high 
implementation campuses, which was the average of the four factors, was 14.1 to 15.6. 
Low implementation campuses had a range of 8.8 to 13.7. 
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Table 6 



Factor Mean Score by Campus 





Understand 

Mean 


Implement 

Mean 


Collaborative/ 

Reflective 

Mean 


Professional 

Development 

Mean 


Campus 

Mean 


High 1 


8.9 


16.4 


17.1 


14.5 


14.2 


High 2 


8.8 


18.0 


18.5 


15.0 


15.1 


High 3 


10.7 


18.8 


18.5 


14.4 


15.6 


High 4 


8.8 


16.4 


17.6 


15.9 


14.7 


High 5 


9.6 


16.0 


16.0 


14.9 


14.1 


High 6 


8.3 


17.4 


18.3 


15.2 


14.8 


Mean-High 


9.2 


17.2 


17.7 


15.0 


14.8 


Low 1 


8.8 


15.1 


16.9 


13.8 


13.7 


Low 2 


8.1 


13.8 


16.5 


13.4 


8.8 


Low 3 


7.9 


16.0 


15.6 


14.9 


13.6 


Low 4 


9.0 


14.6 


13.2 


11.2 


12.0 


Low 5 


8.1 


13.1 


14.7 


12.9 


12.2 


Low 6 


7.3 


11.4 


13.0 


12.4 


11.0 


Mean-Low 


8.2 


14.0 


12.2 


13.1 


11.9 



In every factor except Professional Development, the means showed that the 



principal understood and used the DCM more than the other categories of campus staff. 
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The assistant principal had a higher mean in the Professional Development area than all 
the other categories and was higher on the other three factors when compared to the 
teacher categories (see Table 7). The difference between principal means and the general 
education teacher means was significant in the Understand (15.9 - 8.2 = 7.7) and 
Implement factors (25.9 - 14.7 = 11.2), and less so in the Collaborative/Reflective (20.4 
- 16.0 = 4.4) and Professional Development (16.3 - 13.9 = 2.4) factors. 

Table 7 

Factor Mean Score by Type of Respondent 









Collaborative/ 


Professional 




Understand 


Implement 


Reflective 


Development 


Respondent 


Mean 


Mean 


Mean 


Mean 


Principal 


15.9 


25.9 


20.4 


16.3 


Assistant 










Principal 


14.2 


24.6 


20.2 


18.6 


Instructional 










Specialist 


10.8 


16.7 


16.8 


13.7 


Gen. Ed. 










Teacher 


8.2 


14.7 


16.0 


13.9 


Fine Arts 










Teacher 


6.7 


12.1 


14.6 


12.0 



Note. Factor Mean Score by Campus available in Table 4. 



Factor Research Question 2: Can using mean scores on factors predict TAKS 
passing rates or mean scale score by factors grouping all 12 campuses together? A 
regression analysis was conducted on both the TAKS reading and TAKS math passing 
rates for 2003-2006. However, none of the factors were significant predictors (see Table 



8 ). 
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Table 8 



Regression Analysis Using Campus Mean Scores to Predict TAKS 





Unstandardized 
B SE 


Standardized 

P 


TAKS Reading 


Collaborate 


-0.052 


0.275 


-0.020 


Understand 


-0.259 


0.251 


-0.092 


Implement 


0.277 


0.248 


0.144 


Professional Development 


0.020 


0.314 


0.007 


TAKS Math 


Collaborate 


0.067 


0.277 


0.026 


Understand 


-0.232 


0.252 


-0.082 


Implement 


0.246 


0.249 


0.127 


Professional Development 


-0.091 


0.316 


-0.032 



Note. Sample size = 237 respondents. 



Factor Research Question 3: Did low and high implementation schools score 
differently on the factors? A univariate analysis of variance was performed on the four 
factors to see if there was a significant difference between high and low implementation 
schools (see Table 9). 

As shown, there were significant differences between the high and low 
implementation campuses on all factors except the Understand factor. Most high 
implementation schools had higher mean factor scores than low implementation schools. 
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Table 9 



Univariate Analysis of Variance Measuring Significant Difference Between High and 
Low Implementation Campuses 





N 


df 


MS 


F 


P 




Collaborate 


237 


236 


456 


21.3 


<.001 


0.08 


Understand 


237 


236 


57 


3.1 


.080 




Implement 


237 


236 


657 


17.5 


<.001 


0.07 


Professional Development 


237 


236 


208 


12.0 


.001 


0.05 



Of the survey items, 3 out of 21 asked the respondents to choose one or more of 
the options displayed. Survey items 3, 6, and 7 asked the respondents to choose the 
action, strategy, or tool that they not only understood, but actually used on their campus. 
The other 19 survey items were questions with a Likert scale choice. The following is a 
discussion on the results from survey items 3, 6, and 7. 

Survey item 3 asked if the DCM process helped the teacher/administrator 
understand how to impact instruction. At the high implementation campuses, between 
43% and 60% of the participants who answered said they did understand how to impact 
instruction using the process (see Table 10). At low implementation campuses, between 
20% and 51% of the participants said they understood the process. A total average of 
49% of the high implementation campuses had respondents who said they understood 
how to impact instruction through the DCM process, versus 36% for the low 
implementation campuses. 

Survey item 6 asked if the respondent understood how to use the eight DCM 
tools. Between 20% and 49% of the respondents at the high implementation campuses 
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indicated they knew how to use the tools (see Table 10). At the low implementation 
campuses, only 10% to 31% of the respondents said they felt they knew how to use the 
tools. A total average of 31% of the high implementation campus respondents said they 
actually used the DCM tools versus, 19% for the respondents at the low implementation 
campuses. 

Survey item 7 asked whether the respondents actually used the same eight DCM 
tools. Of the high implementation campuses, between 22% and 41% of the respondents 
said they actually used the DCM tools (see Table 10), while only 10% to 26% of the 
respondents at low implementation campuses actually used the tools. 
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Table 10 

Percentages at the Campuses that Selected the Options Available in Survey Items 3, 6, 



and 7 







Impact 


Understanding 




Usage 




N1 


% 


N2 


% 


N2 


% 


High 1 


144 


60 


63 


32 


58 


29 


High 2 


91 


44 


66 


39 


54 


32 


High 3 


98 


47 


57 


33 


45 


26 


High 4 


92 


52 


62 


43 


59 


41 


High 5 


100 


43 


52 


28 


42 


22 


High 6 


107 


46 


92 


49 


62 


33 


Low 1 


79 


48 


36 


27 


31 


23 


Low 2 


44 


31 


31 


26 


24 


21 


Low 3 


96 


51 


47 


31 


40 


26 


Low 4 


49 


26 


35 


23 


24 


16 


Low 5 


164 


40 


83 


25 


58 


17 


Low 6 


36 


20 


14 


10 


14 


10 



Note. N1 refers to the number of staff who answered the survey multiplied by the total 
number of choices for Survey item 3, which was 11. Percentage (%) refers to the percent 
of options possibly chosen out of the total options available (N). N2 refers to the number 
of staff who answered the survey multiplied by the total number of choices for Survey 



items 6 and 7, which was eight. 
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Survey item 3 offered 1 1 answer choices to show which tools helped participants 
in their implementation of the DCM process. “Sharing data results with peers” (59%) and 
“sharing instructional strategies with peers” (59%) was chosen by participants at 10 out 
of 12 schools as either their first or second choice of most frequently used DCM activities 
(see Table 11). The choice with the next highest percentage was “meeting regularly to 
professionally dialogue” (54%). The answers least selected by participants were “taking 
part in job-embedded professional development” (25%) and “reviewing teacher made 
tests for rigor and relevancy” (19%). 




Table 11 



Respondent Breakdown: Survey Item 3 



Campus 


Analyzing 

TAKS 

Reports 


Analyzing 

Benchmark 

Reports 


Meeting 

Regularly 


Sharing 
Results 
w/ Peers 


Sharing 

Instruct 

Strategy 


Sharing 
Ideas and 
Resources 


Review 
Student 
Work for 
Rigor 


Review 
Teacher 
Made 
Tests for 
Rigor 


Creating 
Action Plan 
for Students 
Based on 
Data 


Taking 
Part in Job- 
Embedded 
Prof. Dev 


Using other 
DCM 
Tools 




# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


High 1 


18 


15.0 


13 


11.1 


16 


12.6 


18 


12.9 


17 


12.2 


14 


12.0 


6 


10.2 


8 


18.2 


14 


16.7 


8 


12.7 


12 


13.2 


High 2 


10 


8.3 


11 


9.4 


11 


8.7 


9 


6.5 


11 


7.9 


7 


6.0 


5 


8.5 


3 


6.8 


7 


8.3 


6 


9.5 


11 


12.1 


High 3 


11 


9.2 


9 


7.7 


14 


11.0 


15 


10.8 


14 


10.1 


11 


9.4 


6 


10.2 


4 


9.1 


6 


7.1 


2 


3.2 


6 


6.6 


High 4 


11 


9.2 


13 


11.1 


7 


5.5 


12 


8.6 


10 


7.2 


9 


7.7 


5 


8.5 


6 


13.6 


6 


7.1 


5 


7.9 


8 


8.8 


High 5 


10 


8.3 


14 


12.0 


11 


8.7 


14 


10.1 


13 


9.4 


15 


12.8 


3 


5.1 


2 


4.5 


7 


8.3 


4 


6.3 


7 


7.7 


High 6 


10 


8.3 


9 


7.7 


16 


12.6 


12 


8.6 


14 


10.1 


9 


7.7 


9 


15.3 


5 


11.4 


8 


9.5 


7 


11.1 


8 


8.8 


Low 1 


9 


7.5 


7 


6.0 


9 


7.1 


9 


6.5 


9 


6.5 


9 


7.7 


4 


6.8 


2 


4.5 


7 


8.3 


7 


11.1 


7 


7.7 


Low 2 


5 


4.2 


6 


5.1 


4 


3.1 


5 


3.6 


6 


4.3 


4 


3.4 


2 


3.4 


2 


4.5 


5 


6.0 


3 


4.8 


2 


2.2 


Low 3 


11 


9.2 


12 


10.3 


10 


7.9 


12 


8.6 


12 


8.6 


9 


7.7 


9 


15.3 


2 


4.5 


7 


8.3 


5 


7.9 


7 


7.7 


Low 4 


4 


3.3 


4 


3.4 


7 


5.5 


8 


5.8 


7 


5.0 


6 


5.1 


0 


0.0 


0 


0.0 


5 


6.0 


3 


4.8 


5 


5.5 


Low 5 


17 


14.2 


18 


15.4 


17 


13.4 


20 


14.4 


21 


15.1 


18 


15.4 


8 


13.6 


9 


20.5 


11 


13.1 


10 


15.9 


15 


16.5 


Low 6 


4 


3.3 


1 


0.9 


5 


3.9 


5 


3.6 


5 


3.6 


1 


0.9 


2 


3.4 


1 


2.3 


1 


1.2 


3 


4.8 


3 


3.3 


Total 


120 


100 


117 


100 


127 


100 


139 


100 


139 


100 


117 


100 


59 


100 


44 


100 


84 


100 


63 


100 


91 


100 


Rank 

Order 


4 




5.5 




3 




1.5 




1.5 




5.5 




10 




11 




8 




9 




7 





U\ 

00 




59 



Of the eight choices offered for survey item 6, the tool selected by the participants 
at the 12 schools almost twice as many times as the next tool was the Classroom Profile 
(68%), followed by the Vertical Alignment Form (37%) (see Table 12). The third highest 
tool selected was the Content Area Guiding Questions (35%), which were sent out by the 
content directors for teachers to reflect on for each early release day. The tools least 
chosen were the Learning Log (27%) and the Cycle of Inquiry (10%). A total average of 
37% of the respondents at the high implementation campuses said they understood how 
to use the eight DCM tools listed, versus 24% for the low implementation campuses. 




Table 12 



Respondent Breakdown: Survey Item 6 





TAKS 

Reflect Tools 


Pre-slugged 

Templates 


Vertical 

Form 


Content Area 
Guiding 
Questions 


Student 

Profile 


Cycle of 
Inquiry 


Classroom 

Profile 


Learn Log 


Total 

Chosen 




# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


High 1 


7 


11.7 


8 


11.1 


8 


9.2 


7 


8.3 


11 


14.1 


1 


4.2 


17 


10.6 


4 


6.3 


63 


9.9 


High 2 


7 


11.7 


11 


15.3 


9 


10.3 


7 


8.3 


8 


10.3 


1 


4.2 


15 


9.4 


8 


12.7 


66 


10.3 


High 3 


10 


16.7 


8 


11.1 


5 


5.7 


9 


10.7 


8 


10.3 


0 


0.0 


14 


8.8 


3 


4.8 


57 


8.9 


High 4 


9 


15.0 


6 


8.3 


10 


11.5 


10 


11.9 


6 


7.7 


3 


12.5 


12 


7.5 


6 


9.5 


62 


9.7 


High 5 


4 


6.7 


6 


8.3 


8 


9.2 


8 


9.5 


7 


9.0 


1 


4.2 


15 


9.4 


3 


4.8 


52 


8.2 


High 6 


7 


11.7 


8 


11.1 


17 


19.5 


11 


13.1 


12 


15.4 


7 


29.2 


19 


11.9 


11 


17.5 


92 


14.4 


Low 1 


3 


5.0 


6 


8.3 


8 


9.2 


6 


7.1 


1 


1.3 


0 


0.0 


8 


5.0 


4 


6.3 


36 


5.6 


Low 2 


2 


3.3 


4 


5.6 


3 


3.4 


4 


4.8 


6 


7.7 


1 


4.2 


7 


4.4 


4 


6.3 


31 


4.9 


Low 3 


6 


10.0 


4 


5.6 


1 


1.1 


7 


8.3 


5 


6.4 


3 


12.5 


15 


9.4 


6 


9.5 


47 


7.4 


Low 4 


2 


3.3 


5 


6.9 


5 


5.7 


4 


4.8 


3 


3.8 


1 


4.2 


12 


7.5 


3 


4.8 


35 


5.5 


Low 5 


11 


18.3 


5 


6.9 


12 


13.8 


10 


11.9 


9 


11.5 


3 


12.5 


23 


14.4 


10 


15.9 


83 


13.0 


Low 6 


2 


3.3 


1 


1.4 


1 


1.1 


1 


1.2 


2 


2.6 


3 


12.5 


3 


1.9 


1 


1.6 


14 


2.2 


Total 


60 


100 


72 


100 


87 


100 


84 


100 


78 


100 


24 


100 


160 


100 


63 


100 


638 


100 


Rank 

Order 


6 




5 




2 




3 




4 




8 




1 




7 









o^ 

o 
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For survey item 7, participants were asked to select the tools that they actually 
used. The Classroom Profile (60%) was again chosen by more teachers and 
administrators than any other (see Table 13), with a similar response rate as in survey 
item 6. Content Area Guiding questions were chosen by 28% and the Pre-slugged 
Templates by 27%. The two least used items were the Learning Log (18%) and the Cycle 
of Inquiry (5%). This was similar to the level of understanding found for survey item 6. 




Table 13 



Respondent Breakdown: Survey Item 7 



TAKS 

Reflection 

Tools 



Pre-slugged 

Templates 



Vertical Form 



Content 

Area 

Guiding 

Questions 





# 


% 


# 


% 


# 


% 


# 


% 


High 1 


7 


11.9 


9 


13.8 


4 


6.3 


5 


8.2 


High 2 


5 


8.5 


8 


12.3 


11 


17.5 


4 


6.6 


High 3 


9 


15.3 


7 


10.8 


3 


4.8 


5 


8.2 


High 4 


9 


15.3 


7 


10.8 


8 


12.7 


10 


16.4 


High 5 


3 


5.1 


5 


7.7 


5 


7.9 


7 


11.5 


High 6 


7 


11.9 


6 


9.2 


11 


17.5 


8 


13.1 


Low 1 


2 


3.4 


4 


6.2 


7 


11.1 


4 


6.6 


Low 2 


3 


5.1 


4 


6.2 


1 


1.6 


2 


3.3 


Low 3 


3 


5.1 


4 


6.2 


1 


1.6 


7 


11.5 


Low 4 


3 


5.1 


3 


4.6 


3 


4.8 


2 


3.3 


Low 5 


6 


10.2 


6 


9.2 


6 


9.5 


7 


11.5 


Low 6 


2 


3.4 


2 


3.1 


3 


4.8 


0 


0.0 


Total 


59 


100 


65 


100 


63 


100 


61 


100 



Rank 

Order 



6 



3 



4 



5 




# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


11 


16.4 


0 


0.0 


18 


12.7 


4 


9.5 


58 


11.4 


7 


10.4 


0 


0.0 


13 


9.2 


6 


14.3 


54 


10.6 


8 


11.9 


0 


0.0 


11 


7.7 


2 


4.8 


45 


8.8 


8 


11.9 


2 


16.7 


10 


7.0 


5 


11.9 


59 


11.5 


6 


9.0 


1 


8.3 


14 


9.9 


1 


2.4 


42 


8.2 


6 


9.0 


4 


33.3 


12 


8.5 


8 


19.0 


62 


12.1 


2 


3.0 


0 


0.0 


10 


7.0 


2 


4.8 


31 


6.1 


3 


4.5 


2 


16.7 


6 


4.2 


3 


7.1 


24 


4.7 


7 


10.4 


2 


16.7 


11 


7.7 


5 


11.9 


40 


7.8 


3 


4.5 


1 


8.3 


8 


5.6 


1 


2.4 


24 


4.7 


6 


9.0 


0 


0.0 


24 


16.9 


3 


7.1 


58 


11.4 


0 


0.0 


0 


0.0 


5 


3.5 


2 


4.8 


14 


2.7 


67 


100 


12 


100 


142 


100 


42 


100 


511 


100 


2 




8 




1 




7 









o^ 

K) 
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The second manner of grouping the survey questions corresponded with the 
research question answered. Their results follow: 

Research Question 1 : What was the relationship for teachers between the self- 
reported (based on the DCM survey results) understanding versus actual use of common 
and continuous assessment? Survey items 4, 6, and 7 were used to answer this question. 
Survey item 4 asked the participant to indicate, on a Likert scale, his or her understanding 
of the need for common and continuous assessments. 

Of the respondents, 56% agreed or strongly agreed that they understood the need 
for common and continuous assessments (see Table 14). When the response option 
“somewhat agreed” was added to the previously mentioned categories, total respondents 
who somewhat agreed to strongly agreed that they understood the need for common and 
continuous assessments increased to 78%. 

Table 14 



Frequency Descriptive of Survey Item 4 





# 


% 


Strongly Disagree 


10 


4.2 


Disagree 


11 


4.6 


Somewhat disagree 


29 


12.2 


Somewhat agree 


52 


21.9 


Agree 


64 


27.0 


Strongly Agree 


69 


29.1 


No Response 


2 


0.8 
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Responses to survey item 6 (use the tool) and 7 (understand the tool) were divided 
into four categories based on the combination of levels of usage and understandings: do 
not use, do not understand; use, do not understand; do not use, understand; use, 
understand. Their ratings of the eight DCM tools are found in Table 15. 

Of the respondents, 53% not only understood the Classroom Profile tool, but 
actually used it as well (see Table 15). Another tool that was used and understood was the 
Pre-Sluggged Template (24.5%). The least understood and used were the Learning Log 
(12.7%) and the Cycle of Inquiry (4%). 

Table 15 

Participants’ Reported Understanding and Uses of the Eight DCM Tools 





Do Not Use, Do 


Use, Do Not 


Do Not Use, 




Use, 




Not Understand 


Understand 


Understand 


Understand 




# 


% 


# 


% 


# 


% 


# 


% 


TAKS Reflection 
Tools 


156 


65.8 


22 


9.3 


11 


4.5 


48 


20.0 


Pre-Slugged 

Templates 


158 


66.7 


14 


5.9 


7 


2.9 


58 


24.5 


Vertical Alignment 
Form 


142 


17.7 


32 


13.5 


8 


3.4 


55 


23.0 


Guiding Questions 


141 


59.5 


35 


14.8 


12 


5.1 


49 


20.6 


Student Profiles 


140 


59.1 


30 


12.7 


19 


8.0 


48 


20.0 


Cycle of Inquiry 


212 


89. 5 


13 


5.5 


1 


0.05 


11 


4.6 


Classroom Profile 


61 


25.7 


34 


14.3 


16 


6.8 


126 


53.0 


Learning Log 


162 


68.4 


33 


13.9 


12 


5.1 


30 


12.7 




65 



In order to present the research and results for Research Questions 2, 3, and 4, the 
following data are offered as a necessary precursor to those questions. In each of these 
questions, TAKS passing rates were assessed by high and low DCM implementation for 
the years 2003-2006. Then, research questions 2, 3, and 4 were analyzed by their unique 
theme, which involved varied survey items. 

TAKS reading passing rates were analyzed using one-way repeated measures 
ANOVA. This analysis revealed no significant between-subjects effect for Usage Type 
(see Table 13). TAKS reading passing rates were significantly higher in 2006 than in 
2003, F (1.58, 370.26) = 507.85, p < .001, = 0.68, but Usage Type had no statistically 

significant effect on the increase in passing rates. 

Across time, there were significant within-group differences from year to year for 
campuses within the high and low implementation groups. For example, some low 
implementation campuses made significant gains, while others in the same group made 
few gains (see Table 16). 

Table 16 



Repeated Measures ANOVA Summary Table for Effect of Usage Type on TAKS Reading 
Percent Passing 





df 


SS 


MS 


F 


P 


2 


Between 














Usage Type 


1.00 


41.05 


41.05 


0.54 


0.46 




Within 


373.42 












Reading 


1.58 


53358.83 


33866.13 


507.85 


0.00 


0.68 


Reading*Usage 


1.58 


869.19 


551.66 


8.27 


0.00 


0.03 


Residual 


310.26 


24691.01 


66.69 








Total 


374.41 
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As shown in Table 17, the high implementation campuses had approximately the 
same steady gain as the low implementation campuses from 2003 to 2005 for TAKS 
reading. A difference appears in the fourth year, with the high implementation campuses 
having a gain of 7.19% passing from 2005 to 2006 versus 2.41% for the low 
implementation campuses. The interaction effect for TAKS reading passing rate and 
Usage Type is significant within groups, which means there were some campuses within 
the group of 12 that did have some significant gains. 

Although the high and low groups differed by only 0.91% (61.44 - 60.53 = 0.91), 
with low implementation campuses having the slightly higher mean in 2003, by the end 
of the fourth year (2006), the high implementation campuses had a mean gain percent 
passing that was 3.66% (83.03 - 79.37 = 3.66) higher. Although that group had strong 
gains from 2003 to 2004 of 11.13% (see Table 17), the increase dropped to 4.18% then 
increased again 7.19% by 2005. The low implementation group had a steady drop in gain 
beginning with 9.36% in 2004, 6.16% in 2005, and 2.41% in 2005. Mean TAKS reading 
percent passing scores from 2003-2006 are also displayed in Table 17. 
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Table 17 

Mean Percent Passing and Mean Gains for TAKS Reading 2003-2006 by Campus Type 





Year 


Mean 


Mean Gains 


High Implementation 


2003 


60.53 




Campuses 


2004 


71.66 


11.13 




2005 


75.84 


4.18 




2006 


83.03 


7.19 



Low Implementation 


2003 


61.44 




Campuses 


2004 


70.80 


9.36 




2005 


76.96 


6.16 




2006 


79.37 


2.41 



The high implementation group’s gains in TAKS reading passing rate were lower 
between 2004 and 2005 (see Figure 1). The significant gain by the high implementation 
campuses can be seen in 2005-2006. 
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TAKS Reading 



Figure 1. Mean TAKS reading percent passing rates, 2003-2006. 



TAKS mathematics passing rates were analyzed using a one-way repeated 
measures ANOVA. This analysis revealed a significance between- subjects effect for 
Usage Type F (1, 235) = 21.59, p < .001, = 0.08 (see Table 18). TAKS math passing 

rates were significantly higher in 2006 than in 2003: F (1.7, 400.21) = 780.6, p < .001, 

T] = 0.77. The effect size of r] = 0.77 indicates that the difference in TAKS math passing 
rate is of high practical significance. There was also an interaction effect with TAKS 
math passing rate and Usage Type. The contrasts of tests within-subjects show that the 
difference in TAKS math passing rate with Usage Types were significant between 



2003-2004 and 2005-2006. 
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Table 18 

Repeated Measures ANOVA Summary Table for Effect of Usage Type on TAKS Math 



Percent Passing 





df 


SS 


MS 


F 


P 


2 


Between 














Usage Type 


1.00 


1940.00 


1940.00 


21.59 


0.00 


0.08 


Within 


403.61 












Math 


1.70 


85322.16 


50100.04 


780.60 


0.00 


0.77 


Math*Usage 


1.70 


1610.91 


945.90 


14.74 


0.00 


0.06 


Residual 


400.21 


25686.17 


64.18 








Total 


404.61 













High implementation campuses showed significant gains from 2003 to 2006. 
Although there was a 3.25% difference in means between each campus group in 2003, 
the gap had widened to 10.1% by 2006 (see Table 19). The interaction effect, “Math and 
Usage,” was significant for gains during the years of 2003-2004 and 2005-2006. 

Low implementation campuses had math gains as well, although not as high as 
the high implementation campuses. Whereas the high implementation group had a higher 
increase than the year before (8.51% gain), the low implementation group had smaller 
gains each year. There was a 13.16% gain from the 2003 to 2004 for the low 
implementation campuses. This was 2.07% less than the high implementation campuses. 
There was a 6.22% gain from 2004 to 2005 and only a 2.72% gain from 2005 to 2006 



(see Table 19). 
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Table 19 



Mean Percent Passing and Mean Gains for TAKS Math 2003-2006 by Campus Type 





Year 


Mean 


Mean Gains 


High Implementation 


2003 


55.90 




Campuses 


2004 


71.13 


15.23 




2005 


76.30 


5.17 




2006 


84.81 


8.51 



Low Implementation 


2003 


52.65 




Campuses 


2004 


65.81 


13.16 




2005 


72.03 


6.22 




2006 


74.75 


2.72 



Although the high implementation group’s math means were always higher than 
low implementation group’s means (see Figure 2), differences in mean percent passing 
rates between the two groups were significant only from 2003 to 2004 and from 2005 to 
2006, as confirmed by the one-way repeated measures ANOVA. The increase in gains 
was not as high between 2004 and 2005, with the high implementation campuses 
showing only a 5.17% gain. That compared to the 15.23% gain the previous year, and an 
8.51% gain from 2005 to 2006. The low implementation group had smaller gains each 



year as shown in Figure 2, from a high of 13.16% in 2004 down to 2.72% in 2006. 
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-♦ — High Implementors 
■■ — Low Implementors 



Figure 2. Mean TAKS math percent passing rates, 2003-2006. 



Research Question 2: What impact does regularly scheduled teacher reflection of 
data have on student achievement? To answer this question, the responses to survey 
item 20, the Collaborative and Reflective factor scores, and Usage Type on TAKS 
reading and math percent passing rates over the years 2003 to 2006 were used. 

When TAKS reading passing rates were analyzed using one-way repeated 
measures ANOVA with survey item 20 and Collaborate as covariates, there were no 
significant between-subject effects (see Table 20). The teachers’ perception of student 
performance accounted for no variation in student achievement scores between high and 



low implementation campuses. 





72 



Table 20 

Repeated Measures ANOVA Summary Table for Ejfect of Usage Type, Perception of 
Student Performance, Survey Item 20, and the Collaboration Factor on TAKS Reading 



Percent Passing 





df 


SS 


MS 


F 


P 


2 


Between 














Collaborate 


1.00 


253.41 


253.41 


3.33 


0.07 




SI_20 


1.00 


168.26 


168.26 


2.21 


0.14 




Usage Type 


1.00 


41.05 


41.05 


0.54 


0.46 




Within 


371.41 












Reading 


1.57 


3909.96 


1303.32 


37.05 


0.00 


0.14 


Reading*Collaborate 


1.57 


63.17 


40.31 


0.60 


0.51 




Reading*SI_20 


1.57 


23.01 


14.68 


0.22 


0.75 




Reading*Usage Type 


1.57 


796.45 


508.23 


7.55 


0.00 


0.03 


Residual 


365.13 


24588.64 


67.34 








Total 


374.41 













Note. Survey Item 20: Answer this question only if your campus uses the DCM: 



Since we began using the DCM, I have seen an improvement in student performance. 



A significant difference was not found in TAKS math passing rates between the 
high and low implementation groups using the Collaborative factor and survey item 20 in 
the between-subject tests (see Table 21). Nor was a significant difference found in the 



within tests in those same math factors. 
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Table 21 

Repeated Measures ANOVA Summary Table for Ejfect of Usage Type, Perception of 
Student Performance, Survey Item 20, and the Collaboration Factor on TAKS Math 



Percent Passing 





df 


SS 


MS 


F 


P 


2 

ri 


Between 














Collaborate 


1.00 


285.01 


285.01 


3.19 


0.08 




SI_20 


1.00 


156.17 


156.17 


1.75 


0.19 




Usage Type 


1.00 


966.46 


1966.46 


22.00 


0.00 


0.09 


Within 


402.98 












Math 


1.70 


7165.88 


4214.39 


65.68 


0.00 


0.22 


Math*Collaborate 


1.70 


144.28 


84.85 


1.32 


0.27 




Math*SI_20 


1.70 


164.06 


96.49 


1.50 


0.23 




Math*Usage Type 


1.70 


1390.14 


817.57 


12.74 


0.00 


0.05 


Residual 


396.18 


25421.92 


64.17 








Total 


405.98 













Research Question 3: What is the relationship between job-embedded 
professional development and student achievement? This research question was answered 
by using the Professional Development factor. This factor was comprised of survey items 
8, 17, 18, and 21, which asked if the respondent felt the DCM tools and strategies learned 
during the collaborative process were impacting student achievement. The Professional 
Development factor did not account for the variance in the dependent variable TAKS 
reading or math passing rates. 

When TAKS reading passing rates were analyzed using one-way repeated 
measures ANOVA with Professional Development as a covariate. Professional 
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Development was non-significant (see Table 22). The only significant within-subject 
effect was for Usage Type, as discussed in the introduction prior to Research Question 2. 
Table 22 

Repeated Measures ANOVA Summary Table for Ejfect of Usage Type, Perception of 
Student Performance, and the Professional Development Factor on TAKS Reading 



Percent Passing 





df 


SS 


MS 


F 


P 


2 

ri 


Between 

Professional 

Development 


1.00 


47.57 


47.57 


0.62 


0.43 




Usage Type 


1.00 


38.59 


38.59 


0.50 


0.48 




Within 

Reading 


373.45 

1.57 


3909.96 


1303.32 


37.05 


0.00 


0.14 


Reading*Professional 

Development 


1.57 


19.89 


19.89 


0.19 


0.78 




Reading*Usage Type 


1.57 


807.85 


807.85 


7.66 


0.00 


0.03 


Residual 

Total 


370.26 

375.45 


24691.01 


66.69 









When TAKS math passing rates were analyzed using one-way repeated measures 
ANOVA with Professional Development as a covariate, Professional Development was 
non-significant (see Table 23). The significant between- subject effect was for Usage 
Type, as discussed in the introduction prior to Research Question 2. The within-subject 
effect was non-significant for Professional Development. 
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Table 23 

Repeated Measures ANOVA Summary Table for Ejfect of Usage Type, Perception of 
Student Performance, and the Professional Development Factor on TAKS Math Percent 
Passing 





df 


SS 


MS 


F 


P 


2 

r[ 


Between 














Professional 














Development 


1.00 


63.72 


63.72 


0.71 


0.40 




Usage Type 


1.00 


2000.02 


2000.02 


22.23 


0.00 


0.09 


Within 


403.23 












Math 


1.70 


7035.54 


4135.14 


64.13 


0.00 


0.22 


Math* Professional 














Development 


1.70 


15.05 


8.85 


1.37 


0.84 




Math*Usage Type 


1.70 


1552.50 


912.48 


14.15 


0.00 


0.06 


Residual 


398.13 


25421.92 


64.48 








Total 


405.23 













Research Question 4: What impact does the use of the DCM have on student 
achievement over a period of 3 years? The data in Table 24 show the increase in the 
percentage of students passing the state assessment (TAKS) in reading beginning the 
second year after TAKS was introduced. The high implementation campuses had 
between a 29% and 71% increase. The low implementation campuses had between a 6% 
loss in percent passing to a 79% increase. The average percent gain for the high 
implementation campuses in reading was 38%, versus 47% for the low implementation 



campuses. 
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Table 24 

Gains and Losses per Campus for TAKS Reading and Math from Spring 2003 to Spring 
2006 





DCM 






Gain 








Three Year Total 
Percent Increase 




Exposure 

Intensity 


2003-2004 


2004-2005 


2005-2006 




Factor 


R 


M 


R 


M 


R 


M 


R% 


M% 


High 1 


42 


19 


19 


13 


8 


0 


3 


71 


65 


High 2 


42 


6 


3 


5 


5 


6 


15 


29 


49 


High 3 


9 


4 


11 


5 


5 


11 


19 


29 


61 


High 4 


6 


9 


14 


1 


10 


8 


6 


27 


58 


High 5 


8 


19 


22 


-10 


3 


12 


2 


30 


30 


High 6 


41 


8 


21 


10 


7 


7 


7 


45 


66 


Low 1 


15 


7 


9 


8 


0 


7 


6 


48 


36 


Low 2 


24 


13 


22 


11 


11 


7 


5 


54 


78 


Low 4 


32 


15 


7 


14 


17 


4 


5 


70 


68 


Low 3 


15 


14 


12 


3 


8 


5 


6 


34 


45 


Low 5 


8 


13 


19 


0 


1 


-8 


5 


-6 


4 


Low 6 


21 


12 


27 


11 


11 


13 


9 


79 


188 


District 


8 


11 


12 


5 


4 


5 


6 


37 


51 


State 


N/A 


7 


7 


3 


5 


4 


4 


19 


27 



Note. Data are based on Panel Recommendation percent passing for each year since 
TAKS began in 2003. R = Reading. M = Math. 
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Table 24 also shows the increases in student performance on the state assessment 
in math beginning the second year after TAKS was introduced. The high implementation 
campuses had between a 30% and 66% increase. The low implementation campuses had 
between a 4% and 79% increase. The average percent gain for the high implementation 
campuses in math was 53%, versus 69% for the low implementation campuses. 

DCM exposure, which was how much extra DCM training a campus received 
over and above the standard district- wide training given to principals each year, was 
documented (see Table 25). Intensity level was calculated by multiplying the number of 
hours a campus had received in addition to the DCM training given to all district 
principals at the beginning of each school year by the intensity factor assigned. The 
length of the training and the audience determined the intensity factor. For example, if an 
all-day training was given to the campus leadership team with the principal present, it 
received a higher intensity level than one-hour campus training. 
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Table 25 



DCM Trainings Available to Campus Professional Personnel during 2004-2006 
Including Intensity Level 





Training Time 


Intensity Level 


1 . Sub-area CILT Meeting 


3 hrs 


4 


2. Sub-area Principal Training 


1 hr 


3 


3. Campus- wide Training 


1 hr 


3 


4. Reading First Coaches' Training 


1 hr 


2 


5. Feeder Pattern Principal Meeting 


1 hr 


2 


6. Meet with Principal/ Admin. Team 


Ihr 


2 


7. INOVA Training 


7 hrs 


5 


8. District-wide Principal Training 


3 hr 


4 



Those campuses with total intensity levels between 6 and 15 had between a 6% 
loss to a 37% gain in reading scores and between 4% to 61% gain in math scores (see 
Table 24). Campuses with total intensity levels between 21 and 42 had 3-year gains 
ranging from 29% to 79% in reading and between 41% and 188% in math. Almost all 
campuses showed strong gains over the 4-year history of the TAKS test, with 6 out of 12 
campuses having 45% or more gains in percent passing in reading and 8 out of 12 in 
math. 

Although all campuses were exposed in some varying degrees to the DCM and all 
campuses showed gains through published passing rates, survey items 14, 16, 17, and 20 
were not shown through statistical tests to impact TAKS gains. These items measured the 
level of implementation of DCM versus TAKS passing rates. Separate one-way repeated 
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measures ANOVA analyses were performed on TAKS passing rates with each survey 
question as a covariate, as seen in Tables 27 through 34. Statistical tests showed the 
interaction effect of passing rates and survey items 14, 16, 17, and 20 to be statistically 
non-significant. Those questions did not significantly account for the variation in gains 
on TAKS passing rates, as can be seen from Tables 27 through 34. 

Survey item 14 asked respondents if they met regularly with peers to collaborate 
on student performance results. The variable did not account for the variance in the 
dependent variable, TAKS reading passing rates. Survey item 14 was non-significant and 
did not account for the variance in TAKS reading passing rates (see Table 26). However, 
the within-subjects contrasts tests were significant, as discussed in the introduction prior 
to Research Question 2. 

Table 26 



Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 14 
on TAKS Reading Percent Passing 





df 


SS 


MS 


F 


P 


Between 












SI_14 


1.00 


121.74 


121.74 


1.59 


0.21 


Usage Type 


1.00 


54.90 


54.90 


0.72 


0.40 


Within 


372.57 










Read 


1.57 


6694.32 


4258.32 


63.55 


0.00 0.21 


Read* SI _14 


1.57 


41.39 


26.33 


0.39 


0.63 


Residual 


367.86 


24649.62 


67.01 






Total 


374.57 











Survey item 16 asked respondents if they regularly implemented the strategies 



learned at the collaborative table. The variable did not account for the variance in the 
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dependent variable, TAKS reading passing rate (see Table 27). However, across time, the 
within-subjects gains in reading were significant, F (1.57, 371.92) = 57.65, p <.001, 
ri^= 0.198. 

Table 27 

Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 16 



on TAKS Reading Percent Passing 





df 


SS 


MS 


F 


P 




Between 














SI_16 


1.00 


128.69 


128.69 


1.69 


0.2 




Usage Type 


1.00 


5.77 


59.77 


1.69 


0.38 




Within 


371.92 












Read 


1.58 


6025.81 


3823.52 


57.65 


0.00 


0.2 


Read* SI _16 


1.58 


230.06 


146.00 


2.20 


0.12 


0.01 


Read*Usage Type 


1.58 


747.04 


474.01 


7.15 


0.00 


0.03 


Residual 


368.78 


24460.96 


66.33 








Total 


373.92 













Teachers’ perceptions of learning at the collaborative table for survey item 17 was 
significant and accounted for differences in TAKS reading passing rates: 

F (1, 373.17) = 2.75, p =.01, ri^= 0.01 (see Table 28). Those with more positive 
perceptions of the effectiveness of the strategies learned at the collaborative table had 
higher TAKS reading passing rates. However, the effect size (rj = .01) indicated that the 
difference was of little practical significance. The within-subjects difference in reading 
was also significant, F (1.58, 370.10) = 48.43, p < .001, = 0.171, but the interaction 



effect of reading with survey item 17 was non-significant. 
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Table 28 

Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 1 7 



on TAKS Reading Percent Passing 





df 


SS 


MS 


F 


P 




Between 














SI_17 


1.00 


208.85 


208.85 


2.8 


0.01 


0.01 


Usage Type 


1.00 


73.78 


73.78 


0.97 


0.33 




Within 


373.17 












Read 


1.58 


5060.65 


3199.68 


48.43 


0.00 


0.17 


Read* SI _17 


1.58 


237.11 


149.91 


2.23 


0.12 




Read*Usage Type 


1.58 


688.71 


435.45 


6.59 


0.00 


0.03 


Residual 


370.10 


24453.91 


66.07 








Total 


373.92 













Survey item 20, which asked if the respondents saw an improvement in student 
performance during the last 3 years, did not account for the variance in the dependent 
variable, TAKS reading passing rates. The variable was non-significant (see Table 29). 
However, the within-subjects contrasts tests were significant, as discussed in the 
introduction prior to Research Question 2. Only the within-subjects gains for reading 
were statistically significant: F (1.60, 371.48) = 96.93, p < .001, = 0.293. 
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Table 29 

Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 20 



on TAKS Reading Percent Passing 





df 


SS 


MS 


F 


P 




Between 














SI_20 


1.00 


16.51 


16.51 


0.22 


0.64 




Usage Type 


1.00 


13.43 


13.43 


0.18 


0.68 




Within 


371.48 












Read 


1.60 


10211.31 


6488.20 


96.93 


0.00 


0.29 


Read* SI _20 


1.60 


39.20 


24.91 


0.37 


0.64 


0.00 


Read*Usage Type 


1.60 


777.20 


493.81 


7.40 


0.00 


0.03 


Residual 


368.28 


24651.81 


66.94 








Total 


373.48 













Survey item 14 asked respondents if they met regularly with peers to collaborate 
on student performance results. The variable did not account for the variance in the 
dependent variable, TAKS math passing rates. Survey item 14 was non- significant (see 
Table 30). The within-subjects contrasts tests were significant, as discussed in the 



introduction prior to Research Question 2. 
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Table 30 

Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 14 



on TAKS Math Percent Passing 





df 


ss 


MS 


F 


P 




Between 














SI_14 


1.00 


178.04 


178.04 


1.99 


0.16 




Usage Type 


1.00 


2114.29 


2114.29 


23.63 


0.00 


0.09 


Within 


400.4 












Math 


1.70 


12356.35 


7283.06 


113.05 


0.00 


0.33 


Math * SI _14 


1.70 


110.19 


64.95 


1.01 


0.36 




Math*Usage Type 


1.70 


1523.73 


898.12 


13.94 


0.00 


0.06 


Residual 


397.0 


25575.98 


64.42 








Total 


402.4 













Survey item 16 asked respondents if they regularly implemented the strategies 
learned at the collaborative table. The variable did not account for the variance in the 



dependent variable TAKS math passing rate. Survey item 16 was non-significant (see 
Table 31). However, across time, the within-subjects gains in math were significant: 

F (1.70, 401.56) = 100.21, p < .001, ri^= 0.30. 
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Table 31 

Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 16 
on TAKS Math Percent Passing 





df 


SS 


MS 


F 


P 




Between 














SI_16 


1.00 


129.28 


129.28 


1.44 


0.23 




Usage Type 


1.00 


2068.03 


2068.03 


23.06 


0.00 


0.09 


Within 


401.56 












Math 


1.70 


10924.87 


6420.61 


100.21 


0.00 


0.30 


Math * SI_16 


1.70 


176.40 


103.67 


1.62 


0.20 




Math*Usage Type 


1.70 


1305.15 


767.04 


11.97 


0.00 


0.05 


Residual 


398.16 


5509.78 


64.07 








Total 


403.56 













Teachers’ perceptions of learning at the collaborative table for survey item 17 was 
significant and did not account for differences in TAKS math passing (see Table 32). 
Those with more positive perceptions of the effectiveness of the strategies learned at the 
collaborative table did not have statistically higher TAKS math passing rates. The within- 
subjects difference in math was significant: F(1.70, 401.89) = 92.95, p <.001, = 0.28. 
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Table 32 

Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 1 7 



on TAKS Math Percent Passing 





df 


SS 


MS 


F 


P 




Between 














SI_17 


1.00 


198.59 


198.59 


2.22 


0.14 




Usage Type 


1.00 


2136.44 


2136.44 


23.90 


0.00 




Within 


401.89 












Math 


1.70 


10169.05 


5971.47 


92.95 


0.00 


0.28 


Math * SI _17 


1.70 


85.64 


50.29 


0.78 


0.44 




Math*Usage Type 


1.70 


1351.66 


793.72 


12.36 


0.00 


0.05 



Residual 

Total 403.89 



Survey item 20, which asked if the respondents saw an improvement in student 
performance during the last 3 years, did not account for the variance in the dependent 
variable, TAKS math passing rates. The variable was non-significant (see Table 33). 
However, the within-subjects contrasts tests were significant, as discussed in the 
introduction prior to Research Question 2. Only the within-subjects for math gains were 
statistically significant: F (1.70, 401.79) = 142.88, p <.001, ri^= 0.38. 
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Table 33 

Repeated Measures ANOVA Summary Table for Effect of Usage Type and Survey Item 20 



on TAKS Math Percent Passing 





df 


SS 


MS 


F 


P 




Between 














SI_20 


1.00 


7.36 


7.36 


0.08 


0.78 




Usage Type 


1.00 


1766.63 


1766.63 


19.58 


0.00 


0.08 


Within 


401.79 












Math 


1.70 


15610.74 


9169.16 


142.88 


0.00 


0.38 


Math *SI_20 


1.70 


39.20 


24.91 


0.37 


0.64 




Math*Usage Type 


1.70 


1360.55 


799.14 


12.45 


0.00 


0.05 


Residual 


398.39 


25566.20 


64.17 








Total 


403.79 













Research Question 5: Does exposure to a model that uses data analysis to impact 
instruction result in a change in the usage of the district’s online data web? The district 
being studied provided campuses with a data web where data reports were housed. 
Administrators and teacher leaders with passwords could look up and print data reports 
after each TAKS and benchmark assessment. The data in Table 34 show the percentage 
of increase in data web pages accessed by the campuses between 2004-2005 and 
2005-2006. 

Data web usage increased for all 12 campuses. The high implementation 
campuses showed gains from 13% to 87% (see Table 34). The campus that showed the 
lowest gain of the high implementers (High 1) had the top usage of the 12 campuses in 
2004-2005 and had the second highest usage in 2005-2006, accounting for its low 



increase. 




87 



Table 34 



Growth by Percentage for 2005 and 2006 Data Web Usage 





Usage 

2005 2006 


Percent Increase 


High 1 


875 


1,002 


13% 


High 2 


129 


1,569 


87% 


High 3 


293 


676 


57% 


High 4 


86 


319 


73% 


High 5 


27 


92 


71% 


High 6 


142 


282 


50% 


Total 


1,552 


3,940 


39% 


Low 1 


554 


1,471 


62% 


Low 2 


840 


951 


12% 


Low 3 


657 


2,162 


70% 


Low 4 


36 


176 


80% 


Low 5 


256 


333 


22% 


Low 6 


138 


1,182 


88% 


Total 


2,481 


6,275 


40% 



The low implementation campuses had increases between 12% and 88%, similar 
to the high implementers (see Table 34). The campus with the 12% growth (Low 6) had 
the highest usage of the six low implementation campuses and dropped to fourth the 
following year. 

The high implementation campuses had accessed 1,552 pages in the data web in 
2004-2005, versus 3,940 pages in 2005-2006. This was a 39% increase from the year 
before. The low implementation campuses increased from 2481 in 2004-2005 to 6,275 
pages in 2005-2006. This was a 40% increase. 
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Of the 230 respondents who answered survey item 5, 50% did not have an 
account that allowed them to access the data web (see Table 35). Of those that answered, 
12% said they used the data web at the “Beginning of the Year and After Each 
Benchmark” and another 12% chose “Weekly.” Of the respondents, 10% said they used 
the web only after each benchmark was given. 

Table 35 



Frequency Descriptive of Survey Item 5 





# 


% 


No Account 


118 


50 


Beginning of the Year Only 


9 


4 


After Each Benchmark 


24 


10 


Beginning of the Year and After Each Benchmark 


29 


12 


Monthly 


21 


9 


Weekly 


29 


12 


No Response 


7 


3 



Research Question 6: Is there a difference in DCM usage perception between 
campus administrators and teachers? A regression was used to measure difference in 
DCM usage perception between campus administrators and teachers using survey items 
3, 6, 7, 8, 9, 13, 15, 16, 19, and 20. There was a significant difference between the 
administrators and teachers in 9 out of 10 questions. Teachers chose more tools or more 
strongly agreed that they implemented the DCM than administrators. Only survey item 
19 (During early release days my campus effectively uses the DCM tools) was non- 
significant, showing no difference between administrators and teachers (see Table 36). 
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Table 36 



Regression Model Showing the Difference in Perception Between Administrators and 
Teachers in DCM Usage Perception 





B 


SE 


P 


f 


DCM Impact (SI 3) 


2.20 


0.8 


0.17* 


0.03 


DCM Know (SI 6) 


2.88 


0.47 


0.36* 


0.15 


DCM Use (SI 7) 


2.5 


0.44 


0.36* 


0.14 


To Help Plan (SIS) 


1.3 


0.40 


0.20* 


0.04 


To Help Guide (SI 9) 


1.9 


0.40 


0.24* 


0.06 


Use Early Release Packet (SI 13) 


3.0 


0.39 


0.45* 


0.25 


DCM Supported (SI 15) 


1.3 


0.41 


0.21* 


0.04 


Implement Strategies (SI 16) 


0.6 


0.39 


O.IO 




DCM during Early Release Day 










(SI 19) 


1.2 


0.42 


0.18* 


0.03 


Improvement in Student 










Performance (SI 20) 


1. 1 


0.41 


0.18* 


0.03 



Note. Variable SI = Survey Item. * = significant at .05 level. 



The regression analysis showed there was a significant difference between 
teachers’ and administrators’ responses with survey item 3 (DCM has helped me 
understand how to impact instruction through: 11 strategies/tools listed). Teachers were 
more apt to choose 2.2 more strategies/tools than administrators (see Table 36). In survey 
item 6 (I understand how to use the following DCM tools: eight strategies/tools listed), 
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teachers chose 2.8 more tools. In survey item 7 (I actually use the following DCM tools: 
eight strategies/tools listed), teachers chose 2.5 more tools than administrators. 

The next seven survey items measured were Likert scale questions. In survey 
items 8, 15, 19, and 20, the teachers agreed one degree more strongly on the scale than 
administrators that they were doing those particular DCM processes. For survey item 16 
(I regularly implement strategies I learn at the collaborative table), there was not a 
measurable difference between teachers and administrators when it came to 
implementing strategies. 

For survey item 9 (I use 1 or more DCM tools to guide my decisions about what 
student interventions to implement), teachers selected this at approximately two degrees 
higher on the Likert scale than administrators. Survey item 13 (I use some or all of the 
Early Release Day packet) had the highest-level difference, with three degrees of 
difference between teachers and administrators. Teachers more strongly agreed that they 
used the Early Release Day packet. 

Research Question 7 : What impact does the principal have on the implementation 
of the DCM model? Survey items 6 and 7 were analyzed by using a cross tabulation to 
compare principals’ understanding and usage of the DCM to teachers’ understanding and 
usage of DCM tools and processes. A tabulation was run on each of the eight tools in 
survey items 6 and 7 (see Table 37). 

In the both the “I understand how to use” and “I actually use” the DCM tools 
categories, principals selected Pre-slugged Templates (see Table 37) as their first choice 
of DCM tool to use (83%). Teachers’ first choice was the Classroom Profile (52%), 
which was the second choice for principals. The next highest percentage for principals 




91 



was 58% with two tools, the TAKS tools and the Vertical Alignment form. There was 
also a tie for second choice between the Pre-slugged Templates and the Guiding 
Questions (20%) for teachers. 

In the “I do not understand how to use” and “I do not actually use” the DCM tools 
category, 50% of the principals and 95% of the teachers selected the Cycle of Inquiry 
(see Table 33). The next highest tool “not used and not understood” was a tie between the 
TAKS tools and the Learning Log (25%) for principals. Of the teachers who responded, 
71% “did not understand or use” the Learning Log and 69% “did not understand or use 
the TAKS tools.” 




Table 37 



Comparing Principals’ and Teachers’ Understanding and Usage ofDCM Tools and Processes 





Use, Understand 




Not Use, 
Understand 






Use, 

Not Understand 






Not Use, 
Not Understand 




Principals 


Teachers 


Principals 


Teachers 


Principals 


Teachers 


Principals 


Teachers 




# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


# 


% 


TAKS Tools 


7 


58 


37 


17 


2 


16 


18 


8 


0 


0 


11 


5 


3 


25 


148 


69 


Pre-slugged Templates 


10 


83 


43 


20 


1 


8 


13 


6 


1 


8 


5 


2 


0 


0 


153 


71 


Vertical Alignment Form 


7 


58 


45 


21 


2 


16 


28 


13 


1 


8 


6 


3 


2 


16 


135 


63 


Content Area Guiding 
Questions 


5 


42 


42 
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Learning Log 
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24 
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25 
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Summary and Chapter Overview 

Chapter 4 presented the results obtained from the DCM survey responses that 
were returned from partieipants. When analyzing the data, survey questions were grouped 
for analysis in three ways. The first way was the selection of high and low 
implementation campuses. The second way was by factors created from subsets of survey 
questions. The third way was by research questions that were answered by factors and by 
survey questions. The chapter contained the results from the survey data that answered 
the questions posed by the way the survey questions were grouped. Various tables and 
figures displayed the statistical results of how the schools were selected, statistical results 
from the four factors, including the factor research sub-questions, and the answers to the 
seven research questions. 

Chapter 5 will present a dialogue about the limitations of the study; the 
implications of the findings, which include interpretation of the data and inferences that 
may be drawn; implications as they relate to the literature; policy implications; and 
suggestions for further studies. The chapter will conclude with a summary of the study. 




Chapter 5 



FINDINGS AND CONCLUSIONS 



This chapter will present a dialogue about the limitations of the study and the 
implications of the findings, which include interpretation of the data and inferences that 
may be drawn. It will also present the implications as they relate to the literature, policy 
implications, and suggestions for further studies. The chapter will conclude with a 
summary and findings of the study. 

The purpose of this study was to examine the relationship between using the Data 
Collaborative Model (DCM), a process of instructional improvement using data analysis, 
and student achievement. The rationale was that campuses, which have attempted to 
create and implement a culture of data-driven decision making in a collaborative, 
reflective setting over a 3-year period of time, would see an increase in teacher 
effectiveness. This process should have a natural output of increased student 
performance. 

This research study was a causal-comparative study, comparing student 
achievement over a 3-year period of time between campuses with a high level of DCM 
implementation and campuses with a low level of DCM implementation. The causal- 
comparative method “is aimed at the discovery of possible causes for the phenomena 
being studied by comparing subjects in whom a characteristic is present with similar 
subjects in whom it is absent or present to a lesser degree” (Borg & Gall, 1989, p. 331). 
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Summary of Research 

Research to this point has been very clear that there are practices known to work 
in urban districts (Council of the Great City Schools, 2002). Well-known researchers are 
voicing more and more firmly that the secret to educational success is known (DuFour, 
2004b; Schmoker, 2004b). Practices such as teacher collaboration, reflective practice, 
professional learning communities, and distributed leadership have been studied and 
found to be successful when implemented consistently (DuFour, 2004b; Fullan, 2001; 
Schmoker, 2004a, 2004b; Timperley, 2005). 

School reform initiatives have been a constant in educational life since the 
National Commission on Excellence in Education published its report in 1983, A Nation 
at Risk: The Imperative for Educational Reform. A variety of reforms have been tried, 
some with limited success, which is dependent on the level of implementation (Eullan, 
2001). The struggle to change high schools through initiatives such as smaller learning 
communities (SECs) is often described in newspapers and educational journals because 
of the interest shown by known personalities such as Bill and Melinda Gates and Warren 
Buffett. 

In Texas, the legal systemic plan for school reform involves these five 
components: (a) a standards-based curriculum known as the Texas Essential Knowledge 
and Skills (TEKS), (b) the state assessment called Texas Assessment of Knowledge and 
Skills (TAKS), (c) an evaluation instrument for principals and teachers (PDAS), (d) a 
public report card of campus and district performance called Academic Excellence 
Indicator System (AEIS), and (e) a state required committee for site-based decision 
making functioning at both the campus and the district level (Texas Association School 




96 



Boards, 2006). In this age of accountability, Texas seems to be leading the way, 
recognizing the importance of principals and teachers by holding them accountable 
through the state assessments and site-based decision making on each campus. 

Because accountability has become such a strong measuring tool, with severe 
consequences for campuses that do not show improvement in student achievement, it has 
become even more important to focus on implementing effective practices that will lead 
to student growth. The researcher designed a study to measure the impact of the Data 
Collaborative Model on a large urban district through focusing on 12 elementary 
campuses selected because of their personnel’s survey responses. 

The age of accountability has focused attention on not only what is taught but also 
how it is taught (Jacob, 1997, 2004; Marzano, Pickering, & Pollock, 2001). Not only is 
there is a need for reliable assessments aligned to rigorous curriculum, but also there is a 
need to know how to use the information gleaned from the assessments (Stiggins, 2004). 
A process must be in place to help campuses understand how to read data and to turn the 
information into an action plan: 

When this process is absent, confusion reigns. Staff from schools indicated that 
they did not see the connection among teacher-administered in-class assessments, 
their norm-referenced district test, and the large-scale state assessment. Nor did 
they know what to do with this information. (Learning Point Associates, 2000, 
p. 1) 

The need to bring a collaborative assessment culture to a campus is becoming 
more and more apparent (Bernhardt, 1999; DuFour & Baker, 1998; Holcomb, 2004; 
Schmoker, 2004b; Stiggins, 2004). The DCM process asked campuses to use a 
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collaborative culture as a way to encourage teachers to meet regularly and to discuss data 
and student work in a non-threatening manner. Encouraging professional dialogue, the 
sharing of strategies, literature studies, shared vision, and distributed leadership are 
components of a professional learning community. The significance of this study 
becomes evident when one takes into consideration the seriousness of the task to be done. 
Implications of Findings 

This part will be divided into three sections. The first section will communicate 
the implications of the survey questions that were grouped in four factors. The second 
section will talk about the implications of the data as shown through the research 
questions. The third section will explore the hypothesis based on the results presented. 

Factors. The factors were formed by sorting the survey items into four themes: 

(a) Understand the DCM, (b) Implement the DCM, (c) Collaborate/ Reflect, and 
(d) Professional Development. After reliability statistics were run, the high 
implementation campuses had factor means that were consistently higher than the low 
implementation campuses in all factors except the Understand factor (see Tables 10 and 
Table 1 1). The Understand factor had the lowest means in all five respondent types 
(principal, assistant principal, instructional specialist, general education teacher, and Fine 
Arts teacher). At high implementation campuses, between 43% and 60% of respondents 
said they understood how to impact instruction through the use of the DCM tools and 
process. Between 20% and 51% of respondents at low implementation campuses 
understood. Campus personnel seemed to be using the tools, even when there was a lack 
of understanding about the process. Principals had the highest means in every factor 
except Professional Development, where the assistant principal mean was 2.3% higher 
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than the principal mean (see Table 10). The difference between principal and general 
education teacher means was significant for the Understand (diff = 7.7) and Implement 
factors (diff = 1 1.2), less so in the Collaborative/Reflective (diff = 4.4) and Professional 
Development (diff = 2.4) factors. This leads the researcher to believe that the principals 
either have not been consistent in their implementation of the DCM or have exaggerated 
their understanding of the DCM for purposes of the survey. There was an even greater 
difference in means between the principal and the Fine Arts teachers, which indicates that 
teachers not teaching the core content areas stand less chance of receiving best practice 
information. 

In analyzing DCM survey items 3, 6, and 7; which asked respondents to select 
which DCM tools and strategies they (a) felt had impacted instruction on their campus, 

(b) understood how to use, and (c) actually used; there was not a significant difference 
between the high and low implementation campuses in two out of the four factors. For 
high implementation campuses, 43%-60% felt like the DCM tools impacted their 
instruction. The low implementation campuses had between 20% and 51% responding 
positively (see Table 10). There were slightly larger differences in the “I know and 
understand how to use the following tools” (survey item 6), with high implementation 
campuses having a range of 28%-49% of respondents indicating understanding. For the 
“I actually use the following tools” (survey item 7), 22%-41% of respondents at high 
implementation campuses and 10%-26% respondents at the low implementation 
campuses indicated usage. Although this lack of significance is reflected throughout the 
results, at the same time, it does begin to indicate that the campuses are beginning to 
understand and use the tools, rather than not using them at all. 
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Hypothesis. The researcher hypothesized that there would be a statistically 
significant difference in TAKS passing rates after using the Data Collaborative Model for 
a 3-year period of time. This was found to be true in TAKS math passing rates, but not in 
TAKS reading passing rates. By the fourth year, 2006, the high implementing group had 
a mean percent gain in reading of 3.66%. The gain in TAKS math was 5.79%. High 
implementing campuses also had higher means than low implementation campuses on 
their perception of DCM’s impact on achievement (survey item 3), understanding the 
DCM tools (survey item), and actually using DCM tools (survey item 7) (see Table 4). 

Research questions. Research Question 1: What was the relationship for teachers 
between the self-reported (based on the survey results) understanding and uses of 
common and continuous assessment? Table 16 reported the percent of respondents that 
“Do Not Use, Do Not Understand” compared to “Use and Understood” by tool and 
strategy type. The larger majority, with ranges of 25.7% to 89.5% of the respondents at 
each campus, indicated that they “Do Not Use, Do Not Understand” the DCM tools. This 
was compared to the “Use and Understand” category with campus percentages ranging 
from 4.6% to 53%. In some cases, up to three times as many people were apt not to 
understand or use a DCM tool or strategy even after 3 years of the DCM being a district 
initiative. This becomes more understandable when one looks at the regularity in 
principal turnover, change of placement in sub-areas done almost yearly, and lack of 
consistent monitoring on the part of the district leadership. Michael Fullan was very clear 
when he said, “general support or endorsement of a new program by itself has very little 
influence on change in practice (e.g. verbal support without implementation follow- 



through)” (Fullan, 2001, p. 81). 
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Research Question 2: What impact does regularly scheduled teacher reflection on 
data have on student achievement? When TAKS reading passing rates were analyzed 
using one-way repeated measures ANOVA with survey item 20 and Collaborate (a 
survey factor) as covariates, there were no significant between- subject effects (see Table 
20). When TAKS math passing rates were analyzed in a similar manner with survey item 
20 and Collaborate as covariates, the only significant between-subject effect was for 
Usage Type. 

Although teachers are beginning to use some of the DCM tools, implementation 
on the campuses is irregular or inconsistent, so significance cannot be attributed 
necessarily to the DCM. It is important to note that there was a significant difference in 
math, which needs to be studied further. Because the district had a math learning 
communities initiative beginning in 2001, there may have been some impact on the math 
data from that. It would need to be studied to verify if this was the case. 

The combined TAKS passing rates and mean gains by high implementation and 
low implementation groups were reviewed. The data showed that the high 
implementation campuses had higher gains between 2003-2006, overall. 

Research Question 3: What is the relationship between job-embedded 
professional development and student achievement? When TAKS reading passing rates 
were analyzed using one-way repeated measures ANOVA with Professional 
Development as a covariate, there were no significant between-subject effects (see Table 
22). The only significant within- subject effect was for Usage Type. 

When TAKS math passing rates were analyzed, between-subject effects for Usage 
Type were significant (see Table 19). The only significant within-subject effect was also 
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for Usage Type. The significance of the within-subject effect may be attributed to the 
inconsistent implementation of the DCM process at campuses. This is reinforced by the 
research done by Joellen Killion (2002). The study on the effectiveness of staff 
development on student achievement mentions that, although sometimes one sees student 
achievement increase after teachers have received staff development, one cannot make an 
immediate assumption that the direct cause of the increase was the staff development: 
Staff development is much like the respiratory system in the body. As one of the 
body systems, it is essential to the body’s basic operation. But, to be fully 
functioning and healthy, the body needs all its systems working together. 

Removal or dysfunction of any system leaves the body in poor health and at risk. 
The same is true for school improvement efforts focused on increasing student 
achievement. To be successful, school improvement requires multiple systems, 
working together to achieve success. These systems include staff development, 
compensation, teacher evaluation, student assessment, and many others. 
Eliminating any one system increases the risk that school improvement efforts 
will be unsuccessful. 

In addition, simply knowing that teachers participated in staff 
development and that student achievement increased does not prove that staff 
development was responsible for the increase. Multiple factors such as higher 
standards, improved curriculum frameworks, and new types of assessment are 
also associated with increased student learning and cannot be measured in 



isolation. (Killion, 2002, p. 22) 
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In the case of the statistical significance found in the between and within test in TAKS 
math Usage Type, one would have to do further study to see what other factors were 
possibly impacting the teacher effectiveness at those campuses. 

The difference in significance between reading and math needs to be studied 
further. One possible answer could be attributed to the math learning communities pilot 
that began in 1998 on a few campuses and has since spread to many campuses in the 
district. If the significance can be traced to that initiative, then it would give credence to 
the research that says that most new initiatives take several years to become sustained in 
the educational environment of the campus of district. As Speck (1996) concluded from 
his research: 

Professional development takes time and must be conducted over several years 
for significant change in educational practices to take place. Substantial change in 
school practice typically takes four to seven years, and in some cases longer. 

(p. 35) 

District administrators must take into account this long time frame. Teachers need to be 
prepared and equipped to engage in professional development throughout their careers. 

Research Question 4: What impact does the use of the DCM have on student 
achievement over a period of 3 years? The data in Table 20 show the increase in the 
percentage of students passing the state assessment (TAKS) in reading beginning the 
second year after TAKS was introduced. The high implementation campuses had 
between a 29% and 71% increase in percent passing. The low implementation campuses 
had between a 6% loss to a 79% increase in percent passing. The average percent gain for 
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the high implementation campuses in reading was 38%, versus 47% for the low 
implementation campuses. 

Increases in student performance on the state assessment (TAKS) in math also 
began the second year after TAKS was introduced (see Table 20). The high 
implementation campuses had between a 30% and 66% increase. The low 
implementation campuses had between a 4% and 79% increase. The average percent gain 
for the high implementation campuses in math was 53%, versus 69% for the low 
implementation campuses. 

Although all campuses were exposed in some varying degrees to the DCM and all 
campuses showed gains on the percent of students passing the TAKS, items measuring 
the level of implementation of DCM (survey items 14, 16, 17, and 20) were not shown 
through statistical tests to impact TAKS gains. Separate one-way repeated measures 
ANOVA analyses by implementation level were performed on TAKS percent passing 
with each survey question as a covariate (see Tables 23 through 30). Statistical tests 
found the difference in math gains were significantly higher for the high implementation 
group than the low; however, gains were not mediated by the concepts measured in 
survey items 14, 16, 17, and 20. 

Research Question 5: Does exposure to a model that uses data analysis to impact 
instruction result in a change in the usage of the district’s online data web? The district 
being studied provided campuses with a data web housing data reports. Administrators 
and teacher leaders with passwords could access and print data reports after each TAKS 
and benchmark assessment. Each time a data report was accessed by someone with a 
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password, it counted as one “page.” Beginning fall 2004, records were kept of campus 
usage of the data web by the amount of data “pages” accessed online. 

Data web usage increased for all 12 campuses. The high implementation 
campuses showed gains from 13% to 87% (see Table 30) in pages accessed. The high 
implementation campus that showed the lowest gain (High 1) had the top usage of the 12 
campuses in 2004-2005 (875 pages) and had the second highest usage in 2005-2006 
(1,002 pages), accounting for its low increase. The low implementation campuses had 
increases between 12% and 88%, similar to the high implementers. The campus with the 
greatest increase was a low implementing campus. It increased in usage from 138 pages 
in 2004 to 1,182 pages in 2006. 

The high implementation campuses had accessed a total of 1,552 pages on the 
data web in 2004-2005, versus 3,940 pages in 2005-2006. This was a 39% increase from 
the year before. The low implementation campuses increased from 2,481 pages in 
2004-2005 to 6,275 pages in 2005-2006. This was a 40% increase. Some of the growth 
percentage in the low implementation campuses may be attributed to the fact that overall 
there were more large campuses in the low implementation group than the high group. 

During the 2004 school year, only campus administrators were allowed to access 
the data on the web. In 2005, the district allowed teacher leaders to have access so they 
could aid in downloading reports for the rest of the campus teachers. This helped explain 
the dramatic increases seen at some campuses. 

Despite the increase in usage of the data web, when respondents were asked how 
often they accessed the data web, only 12% said they did so weekly, another 12% said at 
the beginning of the year and after each benchmark, and 10% said after each benchmark 
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(see Table 35). Only 9% checked the data web monthly, and 4% said they checked the 
data web only at the beginning of the year. The other 50% did not have a password to log 
on to the site. District personnel, both teachers with passwords and administrators, are 
still not in the habit of relying on regular use of the data web to aid them in impacting 
student achievement through data analysis. 

Research Question 6: Is there a difference in DCM usage perception between 
campus administrators and teachers? Out of 10 questions, nine showed a significant 
difference in DCM usage perception between the administrators and teachers. Teachers 
chose more tools or more strongly agreed they implemented the DCM. Only on survey 
item 19 (During early release days my campus effectively uses the DCM tools) did the 
data show there was no significance between administrators and teachers (see Table 36). 
One can conclude that the surveyed campuses are implementing the DCM at the teacher 
level based on teacher responses. In 9 out of 10 questions the principals’ responses did 
not correspond with what the teachers said about implementation of the DCM process 
and tools on their campuses. One could conclude that the principals are not taking an 
active role in the implementation process of the DCM with their teachers. 

The regression analysis found a significant difference between teachers and 
administrators on survey item 3 (DCM has helped me understand how to impact 
instruction through: 11 strategies/tools listed). Teachers were more apt to choose 2.2 
more strategies/tools than administrators (see Table 36). In survey items 6 (I understand 
how to use the following DCM tools) and 7 (I actually use the following DCM tools), 
teachers chose 2.8 and 2.5 more tools, respectively, than administrators. One can 
conclude that, although the teachers are understanding and using the DCM tools as they 
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analyze data, their principals do not necessarily join them. At meetings where DCM tools 
are used, principals may not have the deeper understanding that comes from actual use 
and practice of the process. 

The next seven survey items were Likert scale format. On survey items 8, 15, 19, 
and 20, teachers’ mean scores were slightly higher than administrators regarding using 
certain DCM processes. Although the mean score on survey item 16 was higher for 
teachers, there was not a significant measurable difference between teachers and 
administrators regarding implementing strategies. Teachers do not seem ready for the 
next step of the DCM process, which is to implement the strategies learned at the 
collaborative table. 

For survey item 9 (I use 1 or more DCM tools to guide my decisions about what 
student interventions to implement), the teachers’ mean score was much higher than 
administrators. Survey item 13 (I use some or all of the Early Release Day packet) had 
the greatest difference, where teachers were three times more likely to use the Early 
Release Day Packet than principals (see Table 32). This could be a delegated activity, to 
hand off to teachers the materials to prepare the campus for an early release day, which 
may explain why principals do not use the packet like teachers say they do. 

Although principals were more apt to say they understood and implemented the 
DCM (see Table 7), teachers reported using tools and strategies at a higher rate. This 
could be attributed to principals encouraging teachers to take part in the DCM process, 
but not actually doing the components themselves. 

Research Question 7 : What impact does the principal have on the implementation 
of the DCM model? Eor both “I understand how to use” and “I actually use,” principals 
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chose Pre-slugged Templates (83%) as their first choice of DCM tool to use (see Table 
18). Teachers’ first choice was the Classroom Profile (52%), which was the second 
choice for principals. The next highest percentage for principals was 58% with two tools, 
the TAKS tools and the Vertical Alignment form. There was a tie for second choice for 
teachers between the Pre-slugged Templates and the Guiding Questions (20%). 

In the “I do not understand how to use” and “I do not actually use” category (see 
Table 33), 50% of the principals and 95% of the teachers chose the Cycle of Inquiry. The 
next highest tool not used and not understood was a tie for the principals between the 
TAKS tools and the Learning Log at 25%. Teachers did not understand or use the 
Learning Log (71%), and 69% did not understand or use the TAKS tools, although only 
25% of the principals claimed to not use or understand the same tools. 

There seems to be a dichotomy when it comes to DCM tool usage on the campus. 
Of the principals, 83% claimed they understood and used the Pre-slugged Template, but 
only 20% of teachers said they use the same tool. Among the principals who responded, 
75% said they understood and used the Classroom Profile, but only 52% of teachers did 
the same. Of the principals who responded, 58% said they used and understood the 
TAKS tools, but only 17% of teachers responded similarly. This may be a reflection of 
either principals’ thinking the tools are being understood and used on their campus or an 
exaggeration for purpose of the survey. 

Concluding Discussion 

The large urban district in this study has 217 schools, 160,000 students, and 
20,000 employees. Change comes slowly to any organization (Marzano, Zaffron, Robins, 
Zraik, & Yoon, 1995), but it is especially hard in large bureaucracies. The 12 schools 
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studied reflected the urban trend seen by many large districts (Council of the Great City 
Schools, 2004): 

• Initiatives not monitored to full implementation-not monitored and not supported with 

the resources needed to effect the desired change; 

• High principal turnover-7 of the 12 campuses studied had between two to three 

leadership changes within the 3 years of this study; 

• Changes in sub-areas and/or feeder pattems-6 of the 12 campuses changed sub-area 

and or feeder patterns, which usually meant a change in focus as per the 
superintendent of that sub-area. In half of those cases the sub-area changes occurred 
along with principal changes; 

• General superintendent turnover-the district had two superintendents and an interim 

superintendent during the 3 years of the study. New top leadership impacted 
initiatives and performance targets; and 

• Small gains in student achievement- gains could not be attributed to any one cause of 
practice, due to a lack of consistent implementation in any sub-area or campus. 

During the 3-year study, the district in this study did not have a rigorous and 
relevant curriculum in place long enough to make a difference in scores, nor did it have 
stable leadership, efficient management and strong parental and community support. The 
district had “pockets of excellence” due to sincere campus leaders doing all they could to 
implement effective instructional practices on their campuses. But “stimulating, 
coordinating, and sustaining ‘coherent’ development across many schools is exceedingly 
difficult because it requires balancing top-down and bottom- up forces” (Fullan, 2001, p. 
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170). This district had not learned how to manage those forces with wide success during 
the 3-year study. 

There is researeh pointing to effeetive practices that work with urban sehools. 
Progress is being made toward the performance benchmarks that states have established 
(Council of the Great City Schools, 2004) and it is happening for several different 
reasons. A report written by Council of the Great City Schools (2004) stated that the 
progress can be attributed to, “higher standards; strong and stable leadership; better 
teaching; more instructional time; regular assessments; stronger accountability; parental 
and community support; and efficient management” (p. 65). 

The research gives hope to urban districts. There are practices that can make a 
differenee district- wide, but the implementation process must be well planned and 
systemic. The Data Collaborative Model was a well-intentioned initiative based on 
research and effective practices. It was begun in the spring of 2003 with a benchmark 
template that was created to aid teachers in the disaggregation of their data. By the 
following year, the district leadership created an executive director position to help 
reinforce the initiative. That person was to train 217 campus principals at the beginning 
of the school year and was to create data analysis tools that would aid campuses in 
understanding and using the data being made available to them in the form of TAKS and 
benchmark assessment reports. Although the interim superintendent included the DCM in 
his initiatives list at the beginning of the 2005-2006 school year, little effort was made to 
monitor its implementation through the sub-areas. As Fullan wrote in 2001, “support of 
central administrators is critical for change in district practice. . . .general support or 
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endorsement of a new program by itself has very little influence on change in practice 
(e.g., verbal support without implementation follow through)” (Fullan, 2001, p. 81). 

Examples of lack of unity and support for the DCM initiative included, (a) little 
support and monitoring of implementation at the sub-area level by some of the area 
superintendents; (b) requests for the DCM director to be placed on the sub-area principal 
meeting agendas were often declined or, if accepted, was limited in time; and (c) 
although the district had its own data web, sub-areas often paid large amounts of money 
to outside vendors to bring similar data webs to their campus with no follow-up data 
analysis training. In essence, each sub-area operated as its own domain, with very little 
interference from district leadership. Another example included denying the DCM 
director’s request to build training capacity in instructional coaches and specialists in a 
systemic manner, so that more than one person could be available to train the campuses 
in DCM practices after the TAKS and benchmarks had been given. 

For the district to see change in schools, it must begin with “reculturing towards 
interactive, accountable, inclusive professional learning communities” (Fullan, 2001, 
p. 180). Elmore (2005) also agreed that what is necessary is a change in the organization 
itself, in its very culture. Elmore wrote, “Improvement at scale is largely a property of 
organizations, not of the pre-existing traits of the individual who work in them. 
Organizations that improve do so because they create and nurture agreement on what is 
worth achieving” (p. 25). Organizations must put in place the internal processes by which 
to help people gradually learn how to implement the change in order to achieve the 



desired goal. 
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Recommendations 

For this district to move toward sustainable change that will ultimately result in a 
positive difference in student achievement, it must “reculture” (Fullan, 2001, p. 180) 
itself into an effective learning organization that has the internal commitment from its 
stakeholders to use its “context- specific expertise” (p. 270) to work together to figure out 
what is needed to put in place the needed internal processes mentioned by Elmore (2005). 
Professional learning communities have been the practice that researchers are lauding and 
at the same time lamenting their lack of use (Dufour, 2004b; DuFour & Baker, 1998; 
Schmoker, 2004b). More than one researcher has written forcefully about this research- 
based best practice that has been shown through studies to successfully impact student 
achievement through a change in teacher practice. Schmoker (2004b) spoke for many 
when he wrote. 

But here's the problem. Such "learning communities" — rightly defined — are still 
extremely rare. For years, they have been supplanted and obscured by hugely 
popular, but patently discredited, reform and improvement models. The record is 
clear that these failed, unnecessarily complex reforms have had only the most 
negligible impact on what should be our core concern: the quality of teaching 
students receive. 

As Jim Collins has famously found, any organization attempting 
improvement must first "confront the brutal facts" about itself. In our case, the 
facts point to a fairly stark choice and an unprecedented opportunity for better 
schools. The place to begin is with a hard look at the evidence against 
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conventional reform and improvement efforts — and at the evidence that argues 
for the right kind of "learning communities." (p. 424) 

Policy implications. Among the best practices lauded by researchers such as 
DuFour (2004b) and DuFour, DuFour, Faker, and Karhanek (2004), DuFour and Faker 
(1998), Fullan (2005), Schmoker (2004a, 2004b), and many others is the professional 
learning community where a collaborative culture is set up on a campus. One of the key 
strengths of the professional teaching and learning cycle is its design as a job-embedded 
professional learning process that is ongoing and results driven. According to multiple 
correlation studies on teacher quality (Darling-Hammond, 2000; Darling-Hammond, 
Hightower, Husbands, FaFors, Young, & Christopher, 2003), higher levels of student 
achievement are associated with educators who participate in sustained professional 
development grounded in content- specific pedagogy. Continuous professional learning 
that increases teacher outcomes, in turn, impacts student outcomes. Unfortunately, the 
language in the Texas law that regulates the planning period given to teachers during the 
school day works against the best practice that research says can make a difference in 
teacher effectiveness. The policy stated that: 

. . .each classroom teacher is entitled to at least 450 minutes within each two- week 
period for instructional preparation including parent- teacher conferences, 
evaluating students' homework, and planning. A planning and preparation period 
may not be less than 45 minutes within the instructional day. During that time, a 
teacher may not be required to participate in any other activity. Fducation Code 
21.40. (Texas Association School Boards, 2006, Policy On Fine, 'll 1) 
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The last line of the policy: . .a teacher may not be required to participate in any 

other activity” (Texas Association School Boards, 2006, Policy Online, 'll 1) gives many 
teachers a policy-driven reason not to participate in learning meetings. The language of 
the law is archaic and needs to be updated to reflect the need for teachers to sit down 
regularly together to plan enriched lessons and to keep rigor high in teacher-made 
materials, tests, and student work. 

Suggestions for future research. If the Data Collaborative Model continues to be 
implemented in this district, a similar study of high and low implementation campuses 
should be done after the 6th year of implementation. By then, especially if the Central 
Office leadership actively supports the implementation of the DCM, the difference 
between the high and low campus achievement gains should become more apparent. 

The reason for the difference in statistical significance between the TAKS reading 
and math passing rates needs to be investigated as well. If indeed it turns out that the 
math learning communities initiative, started in 1998, has had such a positive impact, 
then furthering the initiative to include the other content areas should be considered. 
Another way to investigate the significant difference between reading and math would be 
to look at the average scale score by campus in those two content areas. Scale scores are 
more sensitive to smaller changes in scores. Whereas the TAKS passing rates have a two- 
point scale (i.e., pass or fail), the scale scores can discriminate between smaller changes 
due to having many more points on the scale. Looking at the 12 campuses in this study by 
average scale scores might show a change not reflected by passing rates. 

Another area of study that would be valuable is to discern why teachers seem to 
be reluctant to try new strategies that are learned during professional learning 
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communities meetings. This is especially given the fact that the strategies they would be 
implementing are from teachers who are getting higher scores in student achievement. 

Legislators have tended to think of accountability as something new since NCLB, 
but schools have always had some sort of accountability, depending on the policies of 
their district that guides them. What has not changed much within the age of school 
reform is that not much change has actually managed to be sustained. Schools have 
attempted to bring in such practices as professional learning communities, but until we 
begin moving from a culture on which the work of the organization is the sum of the 
work of its individuals to a culture in which individuals’ work is shaped by collective 
expectations, values, and commitments requires the exercise of agency at both the 
individual and collective level (Elmore, 2005). Elmore went on to say that the current 
working model of accountability is flawed. Accountability needs to be the organizational 
response to the needed change, not just action toward compliance or implementation. 

When the district in this study has not only confronted the brutal facts, put in a 
coordinated, well-communicated organizational response to the problems that need to be 
solved, when all district shareholders understand the vision and feel the responsibility to 
work toward shared commitment, then the results will come. Only when teachers feel a 
responsibility not only to their students, but to each other, and show it through sharing 
successful instructional strategies, will campuses get stronger. When principals feel 
responsible not only to their teachers, but to other district principals as well to share their 
strategies with others, will they begin to feel like they are working in the same direction. 
When the Central Office can channel this momentum in a positive way, so that the whole 
school community, including parents and business partners, feels responsibility to helping 
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the district be successful, then schools will begin feeling the positive impact change can 
bring. 

Contained within the process of the Data Collaborative Model are research-based 
best practices that have made other urban schools around the country successful. The 
district in this study should take a second look at the implementation process to see if 
additional resources, time, and attention would benefit its campuses. With the right 
environment, as mentioned above, a process like the Data Collaborative Model could 
flourish and ultimately help students and teachers become stronger learners. 
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Data Collaborative Model 
SURVEY 

1 . Org # 

2. 1 am a: 

□ Principal □ Dean □ Assistant Principal 

□ ESL pull-out, Special Education, or demonstration teacher 

□ Instructional Specialist or Coach on a campus 

□ Classroom Core Teacher (includes ESE/Bilingual teachers) 

□ Eine Arts Teacher □ TAP 

3. Select all that apply: 

The DCM has helped me understand how to impact instruction through: 

□ Analyzing TAKS data reports 

□ Analyzing Benchmark reports with the pre-slugged templates and item analysis reports 

□ Meeting regularly with colleagues to professionally dialogue 

□ Sharing data results with colleagues 

□ Sharing instructional strategies with colleagues 

□ Sharing ideas and resources with colleagues 

□ Reviewing student work for rigor and relevancy 

□ Reviewing teacher-made tests for rigor and relevancy 

□ Creating an action plan for my students based on the data 

□ Taking part in job-embedded professional development opportunities such as sharing 
strategies, observing other teachers, and book/article studies 

□ Using other DCM tools such as the Vertical Alignment Eorm and Classroom Profile 
Benchmark Assessment Eorm 

4. 1 understand the need for common and continuous assessments. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 



TOOES 

5. How often do you access data on the OIR Data Web? 

□ Weekly □ Monthly 

□ At the beginning of the school year only 

□ At the beginning of the school year and after each Benchmark 

□ After each Benchmark only □ I do not have an account 
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6. 1 understand how to use the following DCM tools: 

□ The beginning-of-the-year DCM TAKS reflection tools 

□ The pre-slugged templates that arrive online after each Benchmark 

□ The Vertical Alignment Form 

□ The Content Area Guiding Questions found in the Teacher Packet for Early 
Release Day 

□ The Student Profile document that allows a student to take part in assessment 
feedback and goal setting 

□ The Cycle of Inquiry 

□ Classroom Profile Benchmark Assessment Form 

□ Learning Log that is used to record the ideas and strategies shared during the 
meeting 

7. 1 actually use the following DCM tools: 

□ The beginning-of-the-year DCM TAKS reflection tools 

□ The pre-slugged templates that arrive after each Benchmark 

□ The Vertical Alignment Form 

□ The Content Area Guiding Questions found in the Teacher Packet for Early 
Release Day 

□ The Student Profile document that allows a student to take part in assessment 
feedback and goal setting 

□ The Cycle of Inquiry 

□ Classroom Profile Benchmark Assessment Form 

□ Learning Log that is used to record the ideas and strategies shared during the 
meeting 

8. 1 use one or more DCM tools to help me plan for my professional development. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

9. 1 use one or more DCM tool to guide my decisions about what student interventions to 
implement. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

10. Benchmark results are available online in a timely manner. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

1 1. 1 know how to access the DCM tools online on the OIR Data Web. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

12. 1 know how to access the Early Release Day Principal and Teacher Packet online on 
the OIR Data Web. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

13. 1 use some or all of the Early Release Day Principal and Teacher Packet available 
online on the OIR Data Web. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 
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IMPLEMENTATION 

14. 1 meet regularly with my peers to collaborate on student performance results. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

15. The implementation of the DCM is supported in my school. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

16. 1 regularly implement strategies I learn at the collaborative table. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

17. The strategies I learn at the collaborative table are effective. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

18. 1 feel adequately trained to implement the DCM. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

19. During early release days my campus effectively uses the DCM tools. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

20. Answer this question only if your campus uses the DCM: 

Since we began using the DCM, I have seen an improvement in student performance. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

21.1 feel that I need more training on the OIR Data Web. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 

22. 1 know whom to contact if I have questions about the DCM. 

Strongly Disagree 1 2 3 4 5 6 Strongly Agree 
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Deputy Superintendent E-mail to Administration for DCM Survey Support 
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E-mail from the deputy superintendent of Curriculum and Instruction to the sub- 
area superintendents and the deputy superintendent of school support asking permission 
to send the DCM survey to the campuses. 



»> 




5/1/2006 3:32:05 PM »> 




Evaluation and Accountability and DCM would like feedback on the use of the online 
data and tools available to principals. Below is an e-mail and a web link to a proposed 
survey. Can OIR and DCM have your permission to send this voluntary survey to 
principals for improvement feedback? 

They'd like to solicit improvement input prior to summer staff development. 

Thanks, 




Regards, 

Deputy Superintendent 
Instructional Services 



»»» 








Appendix C 

E-mail Communication to Principals Soliciting Research Participation 
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Copy of e-mail sent to principals and deans of the five participating sub-areas by 
the researcher asking for their participation in the survey. 



»> Becky Good »> 

Your Area Superintendent has given us permission to send your campus the link to the 
OIR/DCM Survey. We will be able meet your data and data analysis needs to more 
accurately through this survey instrument. It should take no longer than 5-7 minutes to 
fill out. 



Please forward this e-mail, which contains the link to the Survey site to all your 
teachers. If e-mail is a problem, please ask your teachers and building administrators to 
access the DCM Survey link on the OIR Data Web page in the Whafs New box. 

Just click on this link of click on the link on the OIR page to get to the survey. 




DCM_Survey/ 



Thanks so much for your valuable input! 
Rebecca Good 
Executive Director, 

Data Collaborative Model 
Instructional Services 




»»» 
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Copy of third e-mail reminder to principals about the DCM survey. 



»> Becky Good 5/22/2006 11:01:23 AM »> 

As we wind down the year, it is important to get your input on the OIR/DCM tools and 
processes to better meet your data analysis needs next year. 

Please remind your teachers and building administrators, as an end of year action, to take 
5 minutes to fill out the short survey. 

Thank you to those whom have already responded! 



Just click on this link or on the link on the OIR page to get to the survey. 
http://^^^^^^^^^DCMSurvey/ 



Thanks so much for your valuable input! 
Rebecca Good 
Executive Director, 

Data Collaborative Model 
Instructional Services 




»»» 
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Letter to Potential Participant 
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Rebecca B. Good 
1724 Allende Ct. 

214-680-8869 
rb-good@comcast.net 
March 28, 2006 
Attention: Principal 
Name of School Goes Here 
Dallas, Texas 75204 
Letter to Potential Participant 

You are requested to participate in a research study on the Data Collaborative Model. It is 
important that everyone who uses the Data Collaborative Model process and the tools 
have an opportunity to give feedback on the effectiveness of their use. By filling this out 
you will help us grow as a district in this instructionally important area. 

In addition to needing this information for district use, Rebecca Good, Executive Director 
of the Data Collaborative Model, will be using this information to write a dissertation on 
the DCM. This important study has implications in all big districts and she thanks you for 
the information that you will be allowing her to use in her doctoral study. This study has 
the approval of the Institutional Review Board at Texas A&M University-Commerce. 
Neither your name nor the name of your school will be used in any reports or 
publications of this study. Should you have questions about this study, contact either 
Tracy Henley (Chair of the Institutional Review Board, Tracy Henley @tamu- 
commerce.edu) , Natalie Henderson, Graduate Office (Natalie Henderson @tamu- 
commerce.edu) via phone at 903-886-5200 or Rebecca Good (researcher) at 214-680- 
8869. Once again, your participation in this study is entirely voluntary. And you may 
withdraw at anytime without penalty. 

The survey will be placed online on the OIR Website. An e-mail will be sent out to the 
campuses to inform everyone when it is online. The survey should take no longer thanlO 
minutes to complete. Your time in answering the survey questions is appreciated. 

Since the survey is online you will not be asked to sign a participants’ agreement form. 

By logging in and answering the questions this researcher understands that you have 
agreed to take part. The survey will come back in an anonymous form. No one will know 
your responses. 

Sincerely, 



Rebecca B. Good 
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E-mail from the deputy superintendent of Curriculum and Instruction to the 
researcher giving permission to send the DCM survey to the campuses. 



»> 




5/1/2006 4:49:59 PM »> 



Permission from for survey. 

I'll send the rest as they come in. When all are received, feel free to send out. 



Regards, 

Deputy Superintendent 
Instructional Services 



»»» 
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